Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
Authors: Yi Zhao
Aidan Scannell
Wenshuai Zhao
Yuxin Hou
Tianyu Cui
Le Chen
Dieter Büchler
Arno Solin
Juho Kannala
Joni Pajarinen
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Reinforcement Learning from offline data
Primary area: reinforcement learning

Abstract: Leveraging offline data is a promising way to improve the sample efficiency of online reinforcement learning (RL).
This paper expands the pool of usable data for offline-to-online RL by leveraging abundant non-curated data that is reward-free, of mixed quality, and collected across multiple embodiments. Although learning a world model appears promising for utilizing such data, we find that naive fine-tuning fails to accelerate RL training on many tasks. Through careful investigation, we attribute this failure to the distributional shift between offline and online data during fine-tuning. To address this issue and effectively use the offline data, we propose two techniques: i) experience rehearsal and ii) execution guidance. With these modifications, the non-curated offline data substantially improves RL's sample efficiency. Under limited sample budgets, our method achieves nearly twice the aggregate score of learning-from-scratch baselines across 72 visuomotor tasks spanning 6 embodiments. On challenging tasks such as locomotion and robotic manipulation, it outperforms prior methods that utilize offline data by a decent margin.

Published as a conference paper at ICLR 2026
EFFICIENT REINFORCEMENT LEARNING BY GUIDING WORLD MODELS WITH NON-CURATED DATA
Yi Zhao" Aidan Scannell 1.2 Wenshuai Zhao Yuxin Hou* Tianyu Cui
Le Chen Dieter Büchler 7,7.8, Arno Solin' 1.3 Juho Kannala 1.10 Joni Pajarinen'
1 Aalto University PUniversity of Edinburgh FELLIS Institute Finland 'Deep Render
Imperial College London Max Planck Institute for Intelligent Systems "CIARR AI Chair
8 University of Alberta
Alberta Machine Intelligence Institute (Amii) Universsiy of Oulu
ABSTRACT
Leveraging offline data is a promising way to improve the sample efficiency of on- line reinforcement learning (RL). This paper expands the pool of usable data for offline-to-online RL by leveraging abundant non-curated data that is reward-free of mixed quality, and collected across multiple embodiments. Although learning a world model appears promising for utilizing such data, we find that naive fine- tuning fails to accelcrate RL training on many tasks. Through careful investiga- tion, we attribute this failure to the distributional shift between offline and online data during fine-tuning. To address this issue and effectively use the offline data, we propose two techniques: i) experience rehearsal and ii) execution guidance. With these modifications, the non-curated offline data substantially improves RL's sample efficiency. Under limited sample budgets, our method achieves nearly twice the aggregate score of ceemiingffrom-seeee tor tasks spanning 6 embodiments. On challenging tasks such as locomotion and robotic manipulation, it outperforms prior methods that utilize offline data by a decent margin.
baselines across 72 visuomo-
INTRODUCTION
Leveraging offline data offers promising way to improve the sample efficiency of reinforcement learning (RL). Prior work has focused primarily on utilizing curated offline data labeled with rewards (Levine al., 2020; Kumar et al., 2020; Fujimoto & Gu, 2021; Kumar et al., 2023), which is expensive and laborious to obtain. For instance, leveraging offline datasets for new robotic manipulation tasks requires retrospectively annotating image-based data with rewards. We instead propose expanding the pool of usable offline data by utilizing abundant non-curated data that is reward-free, of mixed quality, and collected across multiple embodiments. This leads our main research question:
How can we effectively leverage non-curated offline data for efficient RL?
Typical offline-to-online RL methods (Lee et al., 2022; Zhao et al., 2022; Yu & Zhang, 2023; Nakamoto et al., 2024; Nair et al., 2020) fail to utilize non-curated offline data due to their assump- tion of structured data with rewards. While pre-training visual encoders (Schwarzer et al., 2021; Nair al., 2022; Parisi et al., 2022; Xiao al., 2022; Yang & Nachum, 2021; Shang et al., 2024) a common approach to utilize non-curated offline datasets, it fails to fully leverage the rich infor- mation, such as dynamics models, informative states, and action priors for policy learning. On the other hand, learning world models from offline data appears promising for utilizing the non-curated dataset. However, prior work has explored world model training primarily in settings with known rewards (Lu et al., 2023; Rafailov et al., 2023; Hansen et al., 2024) or expert demonstrations (Zhu al., 2024a; Zhou al., 2024; Gao et al., 2025) or focused solely on visual prediction (Zheng al., 2024; Zhu et al., 2024b). Recent approaches (Seo et al., 2022; Wu et al., 2024; 2025) have developed novel architectures for world model pre-training using in-the-wild action-free data, but paid limited
Correspondence www.oaalllaaaoaaaooa to. fi. Code and datasets: https//github.com//haaii 1//ellyiy111////ll

---

Published as a conference paper at ICLR 2026
Sheff
Erc
Enc OOc
Enc Drc
Experience Rehearsal
Place
Cheetah
Run
Execution
Coffee
Guidance
Button
Bax
Cose
Non-Curated Dataset
Agnostg World Model Pre-Training
RL Fine-Tuning
Experience Rehearsal
Guidaane Policy
Fine-Ture Fine-Tuning
TBC
Fine Tune World Model
Enc + Oree
World Model Fine-Tuning
Execution Guidance
Retrieved
2. Augmant Intial States
TRL
Env
Inc
Replay Buffer
Model Rollout
O,)
Figure I: Overview of NCRL (Non-curated offline data for efficient RL). NCRL leverages non- curated offline detadrewar,-freee, mixed-quality, and multi-embodiment- to enable efficient RL. uses this data to pretrain a task-agnostic world model, and then, during fine-tuning, to reduce distributional shift and guide exploration through experience rehearsal and execution guidance.
attention to the fine-tuning process. As result, despite being trained on massive datasets, these wwo---rrooccchh baselines. Additionally, due
methods show only marginal improvements over to the computational costs of RI. experiments, previous work (Wu et al., 2024; 2025) evaluated only on a small set of tasks, leaving the effectiveness of the learned world model unclear on broader tasks. In contrast, we extensively evaluate our method on 72 visuomotor control tasks spanning both loco- motion and robotic manipulation, demonstrating consistent improvements over existing approaches.
Through experiments, we observe that naively fine-tuning a world model fails to improve RL's sample efficiency many tasks. With careful investigation, identify the root cause tributional shift between offline data used for pre-training and online data used for RL fine-tuning. Specifically, when the offline data distribution does not sufficiently of downstream tasks, the pre-trained world models struggle to benefit policy learning due to this distribution mismatch, shown in Fig. 2. Building these insights, offline data in both pre-training and fine-tuning stages, in contrast to previous methods that only consider the offline data for world model pre-training (Wu et al., 2025; Yuan al., 2022; Rajeswar et al.. 2023). To this end, we propose pipeline named Non-curated offline data for efficient RL (NCRL). In the pre-training stage. NCRL learns a task-agnostic world model from non-curated offline data that is reward-free, mix-quality and task-agnostic. In the fine-tuning stage, NCRI. reuses this data through experience rehearsal and execution guidance to mitigate distributional shift by retrieving task-relevant trajectories and to prompt exploration by steering the agent toward regions where the world model has high confidence.
dis-
the data distribution
propose using non-curated
Equipped with our proposed techniques, NCRL demonstrates strong performance across diverse set tasks. Specifically, under a limited sample budget (150k samples), NCRL achieves almost double the aggregate score of learning- merniin-frro--seee from-scratch baselines (DrQ-v2 and DreamerV3), while matching their performance achieved with larger sample budgets. On representative challenging tasks, NCRL outperforms baselines that leverage offline data well state-of-the-art methods us- ing pre-trained world models by a significant margin. Additionally, without any modifications, we show that NCRL improves task adaptation, enabling agents efficiently adapt their skills to new tasks. To summarize, our contributions are:
C1
We propose more realistic setting for leveraging offline data that consists of reward-free and mixed qualitt cobodieembo data.
2

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with different policy learning methods that leverage offline data.
Offline RL
Off2On RL
RLPD
MT Offline RL NCRL (ours)
Reward-free offline data
X
Non-expert offline data
X-embodiment offline data
X
Continual improvement
/
Training stability
C2 We demonstrate that naive world model fine-tuning fails many tasks due to distributional shift between pre-train gre-tranning and fine-tuning data.
C3 We propose two techniques, experience rehearsal and execution guidance, to mitigate the distributional gap and encourage exploration during RL fine-tuning.
C4 We present NCRL, which leverages non-curated offline data in both pre-training and fine- tuning stages and clearly outperforms existing approaches across a diverse set of tasks.
RELATED WORK
In this section, we review RL methods that leverage offline data in different ways. See Sec. for extended discussion and Table for comparison.
RL with task-specific offfine datasets Offline RL trains agents purely from offline data by con- straining divergence from behavior policies (Kumar et al., 2020; Fujimoto Gu, 2021; Kumar al., 2019; Wu et al., 2019; Kostrikov et al., 2021; 2022; Uchendu et al., 2023), but performance depends heavily dataset quality (Yarats et al., 2022). Offline-to-online RI (Lee al., 2022; Zhao al., 2022: Yu & Zhang, 2023; Nair et al., 2020; Rafailov et al., 2023) addresses this by fine-tuning the agent via interaction with the environment. MOTO (Rafailov et al., 2023) proposes model-based RL method with seward-labeled data, but requires model-based value expansion, policy regularization, and controlling epistemic uncertainty conduct stable online RL training- Recent work (Ball et al., 2023; Li et al., 2023) demonstrates promising results by leveraging of- fline data, but still assumes reward-labeles offline data or relies on near-expert data of the target tasks (Li al., 2023), while focus more general setting assuming reward-free -free, mixed mixed-quality and task-agnostic offline data.
RL with multi-task offline datasets Recent work has explored multi-task offline RL (Kumar et al., 2023; Hansen et al., 2024; Julian et al., 2020; Kalashnikov et al., 2021; Yu et al., 2021), but requires known rewards. PWM (Georgiev et al., 2024) trains world model for multi-tast RL but is limited to stte-based inputs and reward-labeled data. To handle unknown rewards, approaches like human labeling (Cabi et al., 2020; Singh et al., 2019), inverse RL (Ng al., 2000; Abbeel & Ng. 2004), or generative adversarial imitation learning (Ho & Ermon, 2016) can be used, though these require human labor or expert demonstrations. Yu al. (2022) assigns zero rewards unlabeled data, which introduces additional bias. Apart from these, there is a line of work that focuses on representation learning or dynamics model training from in-the-wild data (Schwarzer et al., 2021; Parisi et al., 2022; Yang & Nachum, 2021; Yuan et al., 2022; Stooke et al., 2021; Shah & Kumar, 2021: Wang et al., 2022; Sun al., 2023; Ze et al., 2023; Ghosh et al., 2023; Wu et al., 2025; 2023) but fails to utilize rich information in the dataset at the fine-tuning stage.
3 METHODS
In this section, we detail our two-stage approach, which consists of (i) world model pre-training, which learns a multi-task& & embodiment world model, given offline data, which rather importantly, includes reward-free feerdd-freeaee and mixed-quality data, and (ti) RL-based fine-tuning which leverages the pre- trained world model, non-curated offline data, and online interaction an offline- dffline-to-onlig to-lnnine fashion. See Fig. for the overview and Alg. 1 for the full algorithm.
3

---

Published as a conference paper at ICLR 2026
3.1 PROBLEM SETUP
In this paper, we assume the agent has access to a non-curated but in-domain offline dataset Don with three key characteristics: (i) trajectories lack reward labels rf. (ii) data quality is mixed, and (ii) data comes from multiple embodiments. During fine-tuning, the agent interacts with the environment collect labeled trajectories T.n {oi, a rf and stores them in an online dataset Pon (Tin i=1"
Our goal is to learn high-pes perforeeform policy by leveraging both Doff and Don while minimizing the required online interactions Non-
3.2 MULTI-EMBODIMEN WORLD MODEL PRE-TRAINING
During pre-training, rather than training separate models per task as in previous work (Hafner et al., 2020: 2021; 2023), train world model per benchmark and demonstrate that single multi- task & embodiment world model can effectively leverage non-curated data.
Since our primary goal is enabling RL agents use non-curated offline data rather than proposing a new architecture, we adopt the widely-used recurrent state space model (RSSM) (Hafner et al., 2019) with several modifications: (i) removal of task-related losses, (ii) zero-pade sero-padding of actions to unify dimensions across embodiments, and (iii) scaling the model to 280M parameters. With these changes, we show that RSSMs can successfully learn the dynamics of multiple embodiments and can be fine-tundd for various tasks.
Our first stage pre-trains the following components:
Sequence model : ./e = [0(h-1, 21 a1-1) Dynamics predictor 2n ~ (+e he)
Encoder: Et ~ ~0 (2ze 1he oi)
Decoder : ô da her 2t).
The models fa. qo. po and dg are jointly optimized by minimizing:
(()) 1.00 hi) T Σ
(prCp + 32 (z (1)
where $1, B,, B3 are weights of each term. Fpeed minimizes reconstruction error, Layn enables the sequence model and dynamics predictor to predict future latent states, and Ceep encourages the rep- resentation to be more predictable. They are given as:
(0) In (o 22, le)
Layn (f) = max (1, (sg(ooz)) he o:) (ppo (()))
(ep(()) = max (L h4s (:) (a))))
where sg represents the stop gradiettt operator and (L(pllq) is the KL divergence.
While there room to improve world model pre-training through recent self-supe meth- ods (Eysenbach al., 2023) advanced architectures (Vaswani et al., 2017; Gu al., 2022; Mereu et al., 2025), such improvements are orthogonal to our method and left for future work.
3.3 RL-BASED FINE-TUNING WITH REHEARSAL AND GUIDANCE
In our fine-tuning stage. the agent interacts with the environment to collect new data To {oj, a,, ri]r-- This data is used to learn reward function fr re(ft h, 2e) via supervised learning while fine-tuning the world model with Eq. (1). For simplicity, we denote the concatenation of h and predicted by the dynamics predictor pa- The actor and critic are trained using imagined trajectories
= [he, 2,] and use & [hr, &t] when the latent state is
fi = {$f, at I generated by rolling out the policy (a s) with the sequence model fo and the dynamics predictor pg. The rollouts are initialized from states Pu(s) sampled from the replay buffer. The critic (V,,) st) learns to approximate the distribution over the X-retum V,A calculated as:
A)eA + AVA
if f H if t=
7
(3)
A-retum
4

---

Published as a conference paper at ICLR 2026
Distance between Datasets
L-SNE w/o Experience Rehearsal
L-SNE w/ Experience Rehearsal
Averaage Wassersiein Distances
Fandom
offff
dCr
Frotrieved
At the early stage of fine-tuning, there is distribution shift between offline data used for world model pre-training and online data used for RL fine-tuning, which hurts performance. Middle: Experience rehearsal mitigates the distributional shift issue. Right: Quantitatively, at the early stage of fine-tuning, experience rehearsal reduces the Wasserstein distance between the online data and both the offline and expert data.
Figure 2: Visualization of Distribution Mismatch, Left:
where oy El st]] denotes the expectation of the value distribution predicted by the critic. The value function Us is trained by maximizing the log likelihood of the target X-retur, while the actor Te is optimized to maximize the X-retum by backpropagating gradients through the actions and latent states of the imagined trajectories:
H-1
In n (V²) 181) L(Ts) Eeo
H Σ
(-V. 11 H(ae sil) [.1l] (4)
((vs)= E.
Σ
For further details, we refer to DreamerV3 (Hafner et al., 2023)!.
Why Fine-Tuning a World Model Alone is Not Enough? While previous methods typically discard non-curated offline data during fine-tuning (Wu et al., 2025; Rajeswar et al., 2023; Wu et al., 2023), we find that relying solely hard-exploration tasks. To understand why, we analyze the Shelf Place task from Meta- World (Yu al., 2020) world model pre-training and online data Dan collected during early RL training in Fig. 2. The t-SNE plot in Fig. 2 (left) reveals distribution mismatch between Don and Don= leading to three key issues: (i) The world model's accuracy can degrade a significant distributional shift exists between the offline and online data. This degradation is particularly pronounced when the offline data distribution is narrow, which may create a substantial state-space gap. (ii) For hard exploration tasks, the agent struggles reach high-reward regions, causing the world model to be fine-tuned on narrow online data distribution and leading to catastrophic forgetting. (iii) The policy update in Eq. (4) relies imagined trajectories 7 (s(s) I Ta(ai (s)oo(5 st, at), where (s(s) is sampled from Don- A narrow Pu(s) limits the world model to rollout promising trajectories for policy updates. To address these challenges,
pre-trained world model often fails, particularly on
illustrative task by visualizing the distributions of offline data D used for
introduce two key components: i) experience rehearsal, which mitigates distributional shift by retrieving task-relevant trajectories from non-curated datasets (Fig. middle, right), and ii) execution guidance, which encourages exploration by steering the agent toward regions where the world model has high confidence.
Experience Rehearsal Prior works like RLPD (Ball al., 2023) and EXPLORe (Li et al., 2023) have shown that replaying offline data can boost RL training. However, these methods use small, well-structured offline datasets. In our setting, directly replaying non-curated offline data is infeasi- ble since our datasets are 1000 larger and contain diverse tasks and embodiments.
We propose retrieving task-relevant trajectories Pretrieved ( 1.N from the non-curated of- fline data based on neural feature distance between online samples and offline trajectories. This filters out irrelevant trajectories, creating small task-relevant dataset. Specifically, we compute:
D
||o)(ou)) co ce(oor))l2
where is the encoder learned during world model pre-training, and and oof are initial obser- vations from trajectories in the online buffer and offline dataset, respectively. For efficient search to get the top-k similar trajectories, we pre-compute key-value pairs mapping trajectory IDs to neural
We follow the policy update described in the first version of the paper http://////rvv.o/aaa/2220.....10 and Dreamer v2 (Hafner al., 2021).
5

---

Published as a conference paper at ICLR 2026
Aggregate Results 72 Tasks
-5
Walker Run
Cheetah Run Hard
Norma Rzed Score
100
Quadruped Walk
Normalized lized
50
lized
20
50
25
Normalized Scores
200
400
-20
200
400
200
400
Env. Steps (1e3)
Enw. Steps (1e3)
Env. Steps (1e33
Assembly
Shelf Place
Stick Pull
Normalized Score
Normalized Score
100
Normalized Score
221
Mete-World
DMControl
200 Env. Steps (1e3)
400
Env. Steps (1c33
200
400
Env. Steps (1e3)
200
400
NCRL (ours)
ExPLORe
JSRL-BC
UDS
R3M
Figure 3: Left: Quantitative comparison across 72 diverse tasks from Meta-World (Yu et al., 2020) and DMControl (Tassa et al., 2018) with the same sample budget (150k). See Sec. I for full results. Right: Learning curves representative challenging locomotion and robotic manipulation tasks. NCRL consistently outperforms state-of-the-al methods that leverage offline data by a decent margin. We plot the mean and corresponding 95% confidence interval.
features and use Faiss (Douze al., 2024), enabling retrieval in seconds. The retrieval precision can be found in Sec. A.3.
The retrieved data is replayed during fine-tuning, so-called experience rehearsal. The retrieved data serves three purpoes,, as shown in Fig. 1. First, it prevents catastrophic forgetting by continuing to train the world model on relevant pre-training data, particularly important for hard exploration tasks with narrow online data distributions. Second, it augments the initial state distribution (o(s) during model rollout, enabling the world model rollout promising trajectories for policy learning. Third, as described below, it enables learning a policy prior for execution guidance. In Sec. 1. we explain that experience retrieval reduces distribution shift during online fine-tuning. We further demonstrate that experience retrieval acts as a regularizer, helping to prevent catastrophic forgetting during the fine-tuning process. Unlike RLPD and ExPLORe, we do not use this data to learn a Q-function, climinating the need for reward labels.
Execution Guidance via Prior Actors Standard RL training initializes the replay buffer with random actions and collects new data through environment interaction using the training policy. However, offline data often contains valuable information like near-expert trajectories and diverse state-action coverage that should be utilized during fine-tuning. Additionally, distribution shift be- tween offline and online data can degrade pre-trained model weights, making it important to guide the online data collection toward the offline distribution at the early training stage.
To achieve this, we train a prior policy The via behavioral cloning on the retrieved offline data Dretrieved During online data collection, we alternate between this prior policy The and the RL policy To according to a pre-defined schedule. Specifically, at the start of each episode, we probabilisti- cally select whether use The- If The is selected, we randomly choose a starting timestep foR and duration H during which The is active, with T used for the remaining timesteps. In Sec. B.2, we theoretically show that, assuming The outperforms нæ the early stage of training, using a mixed policy composed of The and ne leads to improved policy performance.
While this approach shares similarities with JSRL (Uchendu et al., 2023), our method differs in three key aspects: i) we leverage non-curated rather than task-specific offline data, ii) we demonstrate the benefits a model-based approach over JSRL's model-free framework, and iii) we randomly switch between policies mid-episode rather than only using Tue at episode start. The complete algorithm and theoretical analysis can be found in Sec. and Sec. B, respectively.
4 EXPERIMENTS
In the experiments, we aim answer the following questions:

---

Published as a conference paper at ICLR 2026
Q1 How does NCRL compare to ofate-of-the-ard methods that leverage offline data and train- from-scratch baselines in terms of sample efficiency and final performance?
Q2 How does NCRL compare to other leading model-based approaches that utilize offline data?
Q3 How effectively does NCRL adapt new tasks in a continual learning setting? We further conduct detailed ablation studies to evaluate our method.
Tasks We evaluate our method on pixel-based continuous control tasks from DMControl (Tassa al., 2018) and Meta- Neta-Worll (Yu et al., 2020). The chosen tasks include both locomotion and manipulation tasks covering different challenges in RL, including high-dimensional observations, hard exploration, and complex dynamics. We use three random seeds for each task.
Dataset Our dataset consists of data from two benchmarks: DMControl and Meta-World, visual- ized in Sec. K. For DMControl, we include 10k trajectories covering 5 embodiments collected by unsupervised RL agents (Rajeswar et al., 2023; Pathak et al., 2017), trained via curiosity without task-related information. These trajectories vary in competence and coverage. As the unsuper- vised RL agents are trained to maximize the agent's 's curiosity rather than specific reward signal, the dataset for DMControl does not contain expert trajectories for a specific task (e.g., Walk, Run etc.) For Meta-World, we collect mixed-quality 50k trajectories across 50 tasks using TDMPC- v2 agents (Hansen et al., 2024) by injecting Gaussian noise with o up to 2.0, which intentionally comrppts the policies and produces trajectories of varying success and quality. In practice, such mixture of successful, partially successful, and failed behaviors can naturally arise from, for in- stance, noisy partial human demonstrations collected through telcoperation. In Fig. 9, we assess the dataset quality via imitation learning, showing unsatisfactory performance. This emphasizes the mixed-quality property of the dataset. When combined with the DMControl data, complete offline dataset comprises 60k trajectories (10M state-action pairs) across 6 embodiments.
4.1 NCRL IMPROVES SAMPLE EFFICIENCY ACROSS DIVERSE TASKS
Comparison with Methods that Leverage Offline Data We compare NCRL against several state-of-the-art methods that leverage reward-free data to improve RL training: (i) R3M (Nair et al., 2022), a visual representation pre-training approach that serves as our baseline for comparing pre-trained visual features using non-curated offline data. (ii) UDS-RLPD (Yu et al., 2022; Ball et al., 2023), which assigns zero rewards to offline data and uses RLPD Ball et al. (2023) for policy training. (ii) ExPLORe (Li al., 2023), which labels offline data using UCB rewards. We enhance the original implementation with reward ensembles. (iv) JSRL-BC (Uchendu et al., 2023), which collects online data using a mixture of the training policy and a behavior-cloned prior policy learned from offline data. As the compared baselines cannot handle multi-embodiment data like NCRL, preprocess the offline data to only include task-relevant trajectories for them. Despite the baselines having access to better-structured data, NCRL still significantly outperforms all baselines across the tested tasks. See Sec. G for the details of baselines.
Fig. 3 (right) shows comparison results with baselines. Our method outperforms all compared base- lines by a large margin. Compared to R3M, NCRL shows the importance of world model pre- training and reusing offline data during fine-tuning, versus representation learning alone. R3M fails to improve sample efficiency on most tasks, consistent with findings in Hansen et al. (2023).
UDS and ExPLORe reuse offline data by labeling it with zero rewards and UCB rewards, respec- tively, and concatenating it with online data for off off-policy updates. UDS shows only slightly better performance on Walker Run compared to R3M and JSRL-BC, demonstrating the ineffectivenness of zero-reward labeling. EXPLOORE performs better on 2/3 locomotion tasks and shows progress on challenging manipulation tasks, but NCRL still significantly outperforms it, demonstrating the supe- riority of leveraging pre-trained world model and properly reusing offline data during fine-tuning.
NCRL also clearly outperforms JSRL-BC. JSRL-BC's performance heavily depends on the offline data distribution. While JSRL-BC can perform well when a good prior actor can be extracted from offline data, it struggles with non-expert trajectories, showing only marginal improvements over other baselines on the Cheetah Run Hard, Assembly, and Shelf Place tasks. In contrast, NCRL effectively leverages non-expert offline data. For example, on Quadruped Walk, NCRL benefits from exploratory offline data, enabling pixel-based control within just 100 trials.
7

---

Published as a conference paper at ICLR 2026
Success Rate (%)
100
Button Press
Success Rate (%)
1100
Plate Slide
Success Rate (%)
1.00
Hammer
Success Rate (1)
100
Handle Side
50
50
50
%
60
100
100
%
100
200
Env. Steps (1e3)
Env. Steps (1e3)
Env. Steps (1e3)
Env. Steps (1e3)
NCRL (ours)
|VideoGPT
DreamerV3 (w/ PT)
DreamerV3
DrQ-v2
Figure 4: Comparison with other world model pre-training methods. NCRL outperforms state-
of-the-art model-based methods without relying on techniques used in iVideoGPT, such as reward
shaping and demonstration- based replay buffer initialization.
Stand
Walk
Run
Jump
Roll
Roll Fast
Episode Return
1000
500
NERÉ (aurs)
PackNet
300
600
900 Env. Steps (1e3)
1200
1500
1800
Figure 5: NCRL enables fast task adaptation. We train an RL agent to control an Ant robot from DMControl complete a series of tasks incrementally. NCRL significantly outperforms the widely used baseline PackNet by properly leveraging non-curated offline data.
Methods We compare NCRL with two widely used training- foonn---fro--craacch baselines: DrQ-v2 and DreamerV3, representing model-free and model-based approaches, respectively. Fig. 3 (left) and Sec. I show comparison results on 22 locomotion and 50 robotic manipulation tasks with pixel inputs from DMControl and Meta- World benchmarks. With 150k online samples, NCRL achieves higher aggregate scores compared to DrQ-v2 and DreamerV3, matching their performance obtained with 3.3-6.7x for DMControl, IM for Meta-World), Furthermore, NCRL achieves promising performance on hard exploration tasks where laaaming-from-ccratee baselines fail, such challenging Meta-World manipulation tasks and hard DMControl tasks.
Comparison with Sraining-from-Seree
manipulation
samples (500k
Comparison with Other Model-Based Methods While most tallliittss/muul--mmmmmiiiii models focus on visual prediction (Zheng et al., 2024; Zhu et al., 2024b) or imitation learn- ing (Zhu et al., 2024a; Zhou et al., 2024), some works like Seo et al. (2022), Wu et al. (2024), and VVideoGPT (Wu et al., 2025) investigate world model pre-training with in-the-wild data for RL. These methods typically focus on designing novel or scalable model architectures to leverage the offline data, but lack mechanisms to better leverage offline data during RL fine-tuning. Furthermore, due to the cost RL training, these methods are usually evaluated on limited task sets, making the effectiveness of the pre-traine pre-trained world model unclear on diverse tasks.
world
Figure compares our method with world model pre-training approaches. The baseline results are from the iVieeoGPT paper to get the best reported results in the original paper. We further compare \VideoGPT in an aligned setting in Sec. A.2. Despite extensive pre-training on diverse manipulation data, iVideoGPT and pre-trained DreamerV3 show only marginal improvements scratch baselines. In contrast, NCRL clearly accelerates RL training by properly leveraging non- curated offline data during both pre-training and fine-tuning, Notably, baselines in Fig. 4 reward shaping and expert replay buffer pre-filling, while NCRL uses none of these tricks yet achieves superior performance. This highlights that (i) non-curated offline data contains useful information for RL fine-tuning, and ii) NCRL can effectively leverage such data. Furthermore, NCRL could potentially be combined with iVideoGPT leverage even more diverse offline data in future work.
training- ffom-
4.2 NCRL ENABLES FAST TASK ADAPTATION
We investigate NCRL's benefits for continual task adaptation, where agent must incrementally solve a sequence of tasks. While similar to continual reinforcement learning (CRL) or life-long RL (Parisi et al., 2019; Khetarpal al., 2022),
simplified setting with limited task set.
8

---

Published as a conference paper at ICLR 2026
Walker Run
Cheetah Run Hard
Assembly
Stick Pull
Normalized Score
Normalized Score
40
Normalized Score
100
Normalized Score
60
20
40
50
50
20
-20
200 400 Env. Steps (1e3)
400
400 Env. Steps (2e3)
200
Env. Steps (1e3)
Env. Steps (1e3)
DreamerV3
+P
+P+ER
+P+ER+G (ours)
Figure 6: Ablation study on key components. "P" represents world model pre-training, "ER" means experience rehearsal, and "G" represents execution guidance. The combination of a pre-trained task- agnostic world model with retrieval-baat experience rehearsal and execution guidance boosts RL
performance across diverse tasks.
Dial Turn
Assembly
Stick Pull
Normaliea lizee Score
100
Normalized lized Score
100
Normalized lized Score
100
MM
50
50
50
400
Env. Steps (1e3)
Env. Steps (1e3)
Env. Steps (1e3)
Base (P+ER)
+G (ours)
+OTS
Figure 7: Comparison of execution guidance versus uncertainty- reward labeling. NCRL demonstrates the effectiveness of using execution guidance over uncertainty-based reward label- ing on challenging robotic manipulation tasks.
Note that CRL has a broad scope; assumptions and experiment setups vary among methods, making difficult to set up a fair comparison with other methods. Rather than proposing state-of-the-an -frt CRL method, we aim to demonstrate that NCRL offers an effective approach to leverage previous data that also fits the CRL setting.
Setup & Baselines We set our continual adaptation experiment based on the Quadruped robot from DMControl. Specifically, the agent sequentially learns stand, walk, run, jump, roll, and roll fast tasks with 300K environment steps per task. To have a fair comparison, i.e., having comparable model parameters and eliminating the potential effects from pre-training on other tasks, we pre-train small world model only on the Quadruped domain. During training, the agent can access all pre- vious experiences and model weights. We compare against a widely used baseline PackNet (Mallya & Lazebnik, 2018), which iteratively prunes actor parameters while preserving important weights to remember previous skills. For each new task, PackNet fine-tunes the actor model via iterative pruning while randomly reinitializing the critic model since rewards are not shared among tasks.
Results Figure 5 shows NCRL significaayy outperforms PackNet, enabling adaptation within 100 trials per task. With limited samples, PackNet achieves only 20-60% of NCRL's episodic returns. We attribute NCRL's superior performance to its ability to leverage the diverse offline data through both world model pre-training and fine-tuning with experience rehearsal and execution guidance.
4.3 ABLATIONS
Role of Each Component We now analyze each component's contribution using the same set of tasks from Sec. 4.1. As shown in Fig. 6, world model pre-training shows promising results when the offline data consists of diverse trajectories, such as data collected by exploratory agents (Walker Run), while it fails to work well when the offline data distribution is relatively narrow as in the Meta-World tasks. We found that experience rehearsal and execution guidance stabilize training and improve performance on hard exploration tasks like Cheetah Run Hard and challenging ma- nipulation tasks from Meta- Meta-World. World. This addresses (i) world model pre-training alone, failing to fully leverage rich state and action information from the non-curated offline data and (ii) distribu- tional shift between offline and online data during fine-tuning hurts the learning. The proposed retrieval-base based experience rehearsal and execution guidance help utilize offline data and accelerate exploration, which together enable NCRL to achieve strong performance on a wide range of tasks.
9

---

Published as a conference paper at ICLR 2026
Comparison with Uncertainty-Aware Reward Func- tion To leverage reward-free offline data, ExPLORe (Li et al., 2023) proposes to label offline data with uncertainty- based rewards. To demonstrate the effectiveness of NCRL, compare it with uncertainty-baase instead of using execution guidance, we use Optimistic
Quadruped Walk
Normalized Score
rewards. Specifically,
50
Full
Thompson Sampling (OTS) (Hu al., 2023b) to label the imagined trajectories via model rollout. As shown in Fig. 7,
Enc
Dynamics
our method outperforms the variant using OTS on hard exploration tasks, Assembly and Stick Pull, by large margin,
200 400
Env. Steps (1e3)
showing the effectiveness of using execution guidance.
Figure 8: Impact of fine-tuning dif- ferent world model components.
Comparison of Fine- TuninT Different Components We now investigate the role of different components in the world model during fine-tuning, We use the Quadruped Walk task as a representative task for the investi- gation. As shown Fig. 8, the encoder, decoder, and latent dynamics play important roles during fine-tuning. Fine-tuning the full world model yields the best performance the tested task. The full world model is tnn-ttmed by default in our experiments.
5 CONCLUSION
We propose NCRL, a simple yet efficient approach to leverage ample non-curated offline datasets consisting of reward-free, mixed-quality data collected across multiple embodiments. NCRL pre- trains a task-agnostic world model on the non-curated data and adapts to downstream tasks via RL. We show that naive fine-tuning of world models fails to accelerate RL training due to distributional shift and propose two techniques experience rehearsal and execution guidance to mitigate this is- sue. Equipped with these techniques, we demonstrate that world models pre-trained on non-curated data are able to boost RL's sample efficiency across broader range of locomotion and robotic ma- nipulation tasks. We compared NCRL against a wide set of baselines, including two widely used training-frone
methods, five methods that utilize offline data, and one continual learning method. Our NCRL consistently delivers strong performance over these baselines. Extensive ab- lation studies reveal the effectiveness of the proposed techniques. While promising, NCRL can be improved in multiple ways: extending to real-world applications, leveraging in-the-wild offline data, and exploring novel world model architectures.
ETHICS STATEMENT
This paper contributes to the field of reinforcement learning (RL), with potential applications includ- ing robotics and autonomous machines. While our methods hold promise for advancing technology, they could also be applied in ways that raise ethical concerns, such as autonomous machines ex- ploring the world and making decisions on their own. However, the specific societal impacts of our work are broad and varied, and we believe a detailed discussion of potential negative uses is beyond the scope of this paper. We encourage a broader dialogue on the ethical use of RL technology and its regulation to prevent misuse.
REPRODUCIRILLIT STATRMENT
To facilitate reproducibility, we provide implementation descriptions in Sec. G, report computational requirements in Sec. F, present the complete algorithm in Sec. H, and specify key hyperparameters in Sec. J. Code and datasets are in https///gihub..com//hoy// I/ncrl.
ACKNOWLEDGMENTS
We acknowledge CSC IT Center for Science, Finland, for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through CSC. We acknowledge the computational resources provided by the Aalto Science-IT project. We acknowledge funding from the Research Council of Finland (353138 138, 362407, 352788, 357301, 339730). Aidan Scannell and Wenshuai Zhao were supported by the Research Council of Finland, Flagship program Finnish Center for Artificial Intelligence (FCAI).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning (ICML), 2004.
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chat- topadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. In arXiv preprint arXiv:2501.03577 2025.
Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in in-hand manipulation. The International Journal of Robotics Research (IRR), 2020.
Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 2016.
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. International Conference Machine Learning (ICML), 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. RT-2: vision-language-actto models transfer web knowledge to robotic control. In Conference on Robot Learning (CoRL), 2023a.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. RT-1: Robotics transformer for real-world control at scale. Robotics: Science and Systems (RSS), 2023b.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. International Conference on Learning Representations, 2019.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov. Ksenia Konyushkova, Scott Reed, Rae Jeong. Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning. In Robotics: Science and Sys- tems (RSS), 2020.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lueas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint arXiv:2401.08288 2024.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. International Conference on Learning Representa- tions (ICLR), 2019.
Benjamin Eysenbach, Vivek Myers, Sergey Levine, and Ruslan Salakhutdinov. Contrastive repre- sentations make planning easy. Advances in Neural Information Processing Systems Workshop (NeurIPS Workshop). 2023.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Chongkai Gao, Haozhuo Zhang, Zhixuan Xu, Zhehao Cai, and Lin Shao. FLIP: Flow-centric gen- erative planning for purposs manipulation tasks. International Conference on Learning Representations (ICLR), 2025.
Ignat Georgiev, Varun Giridhar, Nicklas Hansen, and Animesh Garg. PWM: Policy learning with large world models. arXiv preprint arXiv:2407.02466 2024.
Dibya Ghosh, Chethan Anand Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent intentions. In International Conference Machine Learning (ICMI.), 2023.
11

---

Published as a conference paper at ICLR 2026
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. International Conference on Learning Representations (ICLR), 2022.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning (ICML), 2019.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. International Conference on Learning Representations (ICLR), 2020.
Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. International Conference on Learning Representations (ICLR), 2021.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.044 044144, 2023.
Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting a learning-frow scratch baseline. International Conference Machine Learning (ICML), 2023.
Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for con- tinuous control. In International Conference Learning Representations (ICLR), 2024.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Anthony Hu, Lloyd Russell, Hudson Yeo, 7ak Murez, George Fedoseev, Alex Kendall, Jamie Shot- ton, and Gianluca Corrado. GAIA-1: generative world model for autonomous driving. arXiv preprint arXiv:2309. 17080, 2023a.
Bingshan Hu, Tianyue Zhang, Nidhi Hegde, and Mark Schmidt. Optimistic thompson sampling- based algorithms for episodic reinforcement learning. In Uncertainty in Artificial Intelligence (UAI), 2023b.
Ryan Julian, Benjamin Swanson, Gaurav S Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Never stop learning: The effectiveness fine-tuning robotic reinforcement learning. In Conference on Robot Learning (CoRL), 2020.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning (ICML), 2002.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski,
Chelsea Finn, Sergey Levine, and Karol Hausman. MT-OPT: Continuous multi-task robotic rein- forcement learning at scale. arXiv preprint arXiv:2104.082 2021.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty El- lis, et al. DROID: large-scale in-the-wild robot manipulation dataset. Robotics: Science and Systems (RSS), 2024.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforce- ment learning: A review and perspectives. Journal of Artificial Intelligence Research, 2022.
Iyy Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning (ICML). 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. International Conference on Learning Representations (ICLR), 2022.
12

---

Published as a conference paper at ICLR 2026
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Aviral Kumar, Anikait Singh, Frederik Ebert, Mitsuhiko Nakamoto, Yanlai Yang, Chelsea Finn, and Sergey Levine. Pre-training for robots: Offline rl enables learning new tasks from a handful of trials. In Robotics: Science and Systems (RSS). 2023.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Ler- rel Pinto, and Pieter Abbeel. URLB: Unsupervised reinforcement learning benchmark. Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The Handbook of Brain Theory and Neural Networks, 1995.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-tooolinng reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning (CoRL), 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint UrXiv:2005.01643 2020.
Qiyang Li, Jason Zhang, Dibya Ghosh, Amy Zhang, and Sergey Levine. Accelerating exploration with unlabeled prior data. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Advances in Neural Information Processing Systems (NeurIPS), 2021a.
Hao Liu and Pieter Abbeel. APS: Active pretraining with successor features. In International Conference on Machine Learning (ICML), 2021b.
Cong Lu, Philip J Ball, Tim GJ Rudner, Jack Parker- Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. Transactions Machine Learning Research (TMLR), 2023.
Arun Mallya and Svetlana Lazebnik. PackNet: Adding multiple tasks to single network by iterative pruning. In Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, and Sai Rajeswar. Choreog- rapher: Learning and adapting skills in imagination. In International Conference on Learning Representations (ICLR), 2023.
Riccardo Mereu, Aidan Scannell, Yuxin Hou, Yi Zhao, Aditya Jitta, Antonio Dominguez, Luigi Acerbi, Amos Storkey, and Paul Chang. Generative world modelling for humanoids: 1X world model challenge technical report. arxiv preprint arXiv:2510.07092, 2025.
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficien world mod- cls. In International Conference on Learning Representations (ICLR), 2023.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.099: 2020.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinay Gupta. R3M: A uni- versal visual representation for robot manipulation. In Conference on Robot Learning (CoRL), 2022.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-QL: Calibrated offline pre- pre-training for efficient online fine- tuning. In Advances in Neural Information Processing Systems (NeurIPS). 2024.
13

---

Published as a conference paper at ICLR 2026
Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning (ICML), 2000.
Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, al. Open X- Embodiment: Robotic learning datasets and rt-x models. In International Conference on Robolics and Automation (ICRA), 2023.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. In International Conference on Machine Learning (ICML), 2022.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning (ICML), 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International Conference on Machine Learning (ICML). 2019.
Tim Pearce, Tabish Rashid, Dave Bignell, Raluca Georgescu, Sam Devlin, and Katja Hofmann. Scaling laws for pre-training agents and world models. arXiv preprini arXiv:2411.04434 2024.
Rafael Rafailov, Kyle Beltran Hatch, Victor Kolev, John D Martin, Mariano Phiclipp, and Chelsea Finn. MOTO: Offline pre-training to online tne-tuning for model-based robot learning. In Con- ference Robot Learning (CoRL), 2023.
Sai Rajeswar, Pietro Mazzaglia, Tim Verbelen, Alexandre Piché, Bart Dhoedt, Aaron Courville, and Alexandre Lacoste. Mastering the unsupervised reinforcement learning benchmark from pixels. In International Conference on Machine Learning (ICML), 2023.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. Transactions on Machine Learning Research (TMLR), 2022.
Aidan Scannell, Mohammadreza Nakhaei, Kalle Kujanpää, Yi Zhao, Kevin Luck, Aroo Solin, and Joni Pajarinen. Discrete codebook world models for continuous control. In International Confer- ence on Learning Representations (ICLR), 2025.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R De- von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Ramanan Sekar. Oleh Rybkin, Kostas Danilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning (ICMI.), 2020.
Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-fred free pre-training from videos. In International Conference on Machine Learning, pp- 19561-19579. PMLR, 2022.
Rutav Shah and Vikash Kumar. RRL: Resnet as representation for reinforcement learning, In International Conference on Machine Learning (ICML), 2021.
Jinghuan Shang, Karl Schmeckpeper, Brandon B May, Maria Vittoria Minniti, Tarik Kelestemur, David Watkins, and Laura Herlant. Theia: Distilling diverse vision foundation models for robot learning. In Conference on Robot Learning (CoRL), 2024.
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. Robotics: Science and Systems (RSS), 2019.
Adam Stooke, Kimin Lee, Pieter Abbecl, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference Machine Learning (ICMI.), 2021.
14

---

Published as a conference paper at ICLR 2026
Yanchao Sun, Shuang Ma, Ratnesh Madaan, Rogerio Bonatti, Furong Huang, and Ashish Kapoor. SMART: supervipd multi-task pretraining with control transformers. In International Con- ference Learning Representations, 2023.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801 006900 2018.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees. Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. Robotics: Science and Systems (RSS), 2024.
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. In Inter- national Conference on Machine Learning (ICML), 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- lion Processing Systems (NeurIPS), 2017.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 2019.
Che Wang. Xufang Luo, Keith Ross, and Dongsheng Li. VRL3: A data-driven framework for visual deep reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Jialong Wu, Haoyu Ma, Chaoyi Deng, and Mingsheng Long. Pre-training contextualized world models with in-the-will wild videos for reinforcement learning. Advances in Neural Information Pro- cessing Systems, 36:39719-39777 2023.
Jialong Wu, Haoyu Ma, Chaoyi Deng, and Mingsheng Long. Pre-training contextualized world models with in-the-wild videos for reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. iVideoGPT: Interactive videogpts are scalable world models. Advances in Neural Information Processing Systems (NeurIPS), 2025.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 2019.
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06177 06173, 2022.
Yingchen Xu, Jack Parker-Holder, Aldo Pacchiano, Philip Ball, Oleh Rybkin, S Roberts, Tim Rocktäschel, and Edward Grefenstette. Learning general world models in handful reward-free deployments. In Advances in Neural Information Processing Systems (NeurIPS), 2022..
Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential deci- sion making. In International Conference on Machine Learning (ICML), 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. In International Conference on Machine Learning (ICML). 2021.
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv: 13425, 2022.
Tianhe Yu, Deirdre Quillen. Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- World: A benchmark and evaluation for multi-task and meta reinforcement learn- ing. In Conference Robot Learning (CoRI.), 2020.
15

---

Published as a conference paper at ICLR 2026
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. In International Conference on Machine Learning (ICML), 2022.
Zishun Yu and Xinhua Zhang. Actor-critic alignment for offline-to-onling reinforcement learning. In International Conference on Machine Learning (ICML), 2023.
Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang. Yi Wu, Yang Gao, and Huazhe Xu. Pre- trained image encoder for generalizable visual reinforcement learning. In Advances in Neural Information Processing Systems (NeurlPS), 2022.
Yanjie Ze, Nicklas Hansen, Yinbo Chen, Mohit Jain, and Xiaolong Wang. Visual reinforcement learning with self-supervised 3d representations. Robotics and Automation Letters, 2023,
Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behav- ior cloning regularization for stable offline-to-online reinforcement learning. arXiv preprint MrXiv:2210.13846 2022.
Yi Zhao, Le Chen, Jan Schneider, Quankai Gao, Juho Kannala, Bernhard Schölkopf, Joni Pajari- nen, and Dieter Büchler. RPIM: large-scale motion dataset for piano playing with bi-manual dexterous robot hands. Conference on Robot Learning (CoRI.), 2024.
Zangwei Zheng, Xiangyu Peng. Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-Sora: Democratizing efficient video production for all. arXiv preprint arXiv:24/2 12.20444, 2024.
Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. RoboDreamer: Learning compositional world models for robot imagination. International Conference Ma- chine Learning (ICML), 2024.
Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong, IRASim: Learn- ing interactive real-robot action simulators. arXiv preprint arXiv:2406. 14540, 2024a.
Zheng Zhu, Xiaofeng Wang. Wangbo Zhao, Chen Min, Nianchen Deng, Min Dou, Yuqi Wang, Bo- tian Shi, Kai Wang, Chi Zhang, et al. Is sora world simulator? a comprehensive survey on general world models and beyond, IEEE Transactions on Pattern Analysis and Machine Intelli- gence (TPAMI), 2024b.
16

---

Published as a conference paper at ICLR 2026
Appendices
A More Results
18
Theoretical Analysis
21
C More Related Work
23
D Limitations
24
Disclosure of LLMs Usage
24
Compute Resources
24
G Implementation Details
24
Algorithm
27
I Full Results
28
J Hyperparameters
34
Task Visualization
35
17

---

Published as a conference paper at ICLR 2026
A MORE RESULTS
A.1 COMPARISON WITH IMITATION LEARNING BASELINE
To demonstrate the mixed-quality property of the non-curated dataset, compare NCRL with Diffusion Policy, a widely used imitation learning approach by modeling the agent with diffusion models. From Fig. we can see that due to the dataset consisting of non-expert data, the diffusion policy fails to demonstrate satisfactory results, while NCRL can effectively utilize the offline data.
Normalized Score
Walker Run
Normalized Score
40
Cheetah Run Hard
Normalized Score
100
Quadruped Walk
60
20
40
50
20
-20
200
400
400
200
400
Env. Steps (1e3)
Env. Steps (1e3)
Env. Steps (1e3)
Assembly
Shelf Place
Stick Pull
Normalized Score
100
Normalized Score
100
Normalized Score
100
50
50
50
400
200
400
Env. Steps (1e3)
Env. Steps (1e3)
Env. Steps (1e3)
NCRL (ours)
Diffusion Policy
Figure 9: Comparison with Diffusion Policy. NCRL can effectively handle non-curated offline data while the imitation learning baseline fails.
A.2 COMPARISON WITH VIDEOGGPT
Comparison in an Aligned Setting
In Fig. we compare our method against the original iVideoGPT results (Wu et al., 2025).
Performance Comparison Across Tasks
Their experimental setups differ than ours in several ways: i) iVideoGPT modifies the re- ward function to assign high rewards to suc- cessful episodes and ti) pre-fills the replay buffer with a few demonstrations to ease explo- ration. In addition, iVideoGPT's is pre-trained X-embodiment datasets (O (O'Neill et al., 2023), whereas our method uses data from the same domain as the downstream tasks.
Suceess Rate
0.5
Slide
Assembly
Press
Pull Side
NCRL (ours)
MVideoGPT-allgned
Figure 10: Comparison with aligned iVideoGPT.
To control for these differences,
additional set of experiments. Specifically, fine- tune iVideoGPT on our dataset, initialize the policy with behavior cloning, and remove both reward shaping and demonstration pre-filling. We refer to this variant as iVideoGPT-align. We compare VVideoGPT--llggn with our NCRL after training with 200k environment steps Fig. 10, NCRL still outperforms (VideoGPT-align with a decent margin.
Full Results of Comparison with IVideoGPT We compare with other model- based approaches on tasks used in iVideoGPT (Wu et al., 2025). We show that NCRL outperforms the baselines without using reward shaping and pre-filling the replay buffer with demonstrations. This highlights that although non-curated, the offline data can clearly boost RL training, and NCRL can effectively use the information in the data.
A.3 EXPERIENCE RETRIEVAL PERFORMANCE
In Sec. 3.3, we adopt a simple criterion to retrieve task-relevant trajectories from the non-curated dataset. We evaluate retrieval performance in Table 2, reporting precision the top-250 and top-
18

---

Published as a conference paper at ICLR 2026
Button Press TW.
Plate Slide
Hammer
SuccKs Rate (N0
100
100
50
50
50
40
0
50
100
50
100
Env. Staps (x10')
Env. Staps (x10")
Env. Staps (x10")
Handle Side
Coffee Push
Sucross Rata (50)
100
100
100
50
100
%
100
200
200
400
Env. Staps (x107)
Env. Staps (x10")
Env. Staps (x107)
NCRL (ours)
!VideGPPT
DreamerV3 (w/ PT)
Dreamerv3
DIQ+2
Figure 11: Comparison with model-based approaches for leveraging offline data.
500 retrieved trajectories. Our method achieves consistently high precision. For the Door Open task, some retrieved trajectories overlap with related tasks (Door Close, Door Lock, Door Unlock), but find that RL training remains effective across all 72 evaluated tasks. A likely reason that most RL training data is collected online, with policy and value functions updated from imaginary data generated by model rollouts, which mitigates the impact of occasional aask-irreeevant trajectories. We expect future work to explore more advanced retrieval strategies for improved robustness.
Table 2: Precision results across tasks.
Tasks
Quadruped Run Assembly
Shelf Place
Door Open
Precision @ 250
100% 100%
100%
100% 100%
84%
Precision @ 500
100%
68%
A.4 MORE ABLATION STUDIES
Hyperparameter Sensitivity In execution guidance, we randomly sample both the starting timestep faant and duration H. Unlike JSRL (Uchendu et al., 2023), our approach eliminates pensive tuning for these hyperparameters and demonstrates robust performance. In this stage, we only introduce one hyperparameter to probabilistically decide whether use тBC based on linear annealing schedule. We show that in Fig. 12, our method is not sensitive to this annealing schedule, showing robustness wide range of possible schedules.
Assembly
Quadruped Walk
Normalized Score
100
Noraltzed Score
100
50
50
200
400
200
400
Env. Steps (1
Enw. Steps (1e3)
linear(1,0,0)
linear(1,0,50k)
linear(1,0,150k)
linear(1.0,250k)
Figure 12: Our method is less sensitive to the choice of the execution guidance annealing schedule.
Role of Each Component We show inter-quartile mean (IQM) and optimality gap for the abla- tion study of the role of each proposed component in Fig. 13. Together with the retrieval-based experience rehearsal and execution guidance, a pre-trained task-agnostic world model boosts RI. performance on wide range of tasks.
Impact of Retrieved Data In Fig. 14, we evaluate the impact of retrieved data on the agent's performance to assess the robustness of NCRL with respect to the quality of the retrieved dataset. As shown in the experiments on three challenging MetaWorld manipulation tasks, we progressively trajectories with 0%, 25%, 50%, 75%, and 100% trajectories
replaced the retrieved that lie far from the target task in the latent space. We observed that our method remains robust even the quality of the retrieved data degrades.
19

---

Published as a conference paper at ICLR 2026
IQM Normalizz lized Score
DMControl tasks
MetaWorld tasks
1.0
IQM
Optimality Gap
DreamerV3
0.5
+P+ER
+P+ER+G (ours)
0.1
0.5
0.9
0.4 0.6
0.8
200
200 Env. Steps (1e3)
Normalized Score
Steps (1e3)
Figure 13: Ablation study on the role of each component. "P" represents world model pretraining, "ER" means experience rehearsal, and "G" represents execution guidance. Together with the pro- posed retrieval- experience rehearsal and execution guidance, world model pre-training boosts RL performance on wide range of tasks.
Shelf Place
Assembly
Stick Pull
Normalized Score
100
Normalized alized Score
100
Normalized ized Score
100
50
50
50
200 Env. Steps (1e3)
Env. Steps (1e3)
Env. Steps (1e3)
0% Task-Irrelevant
50% Task-Irrelevant
100% Task-Irrelevant
25% Task-Irrelevant Data
75% Task-Irrelevant Data
Figure 14: Comparison with injecting different ratios of irsk-irrelevant offline data. Our method remains robust even as the quality of the retrieved data degrades.
A.5 MODEL SIZE OF DREAMERV3
In Sec. I, we compare NCRL with the Dreamer baseline under a commonly used but relatively small model-size configuration. Although DreamerV3 has shown performance gains on more chal- lenging domains such as Craft and DMLab when using larger models, these benefits are less pro- nounced the settings examined in this work (DMControl and MetaWorld). Indeed, DreamerV3 itself uses a relatively small model for DMControl tasks Hafner al. (2023). To ensure a fair com- parison, we additionally evaluated D3eamerV3 using the same model size as NCRL. As shown in Fig. 15, increasing the model size improves DreamerV3's performance Walker Run but degrades performance on Quadruped Walk. However, NCRL consistently outperforms the DreamerV3 base- line across different model sizes.
Quadruped Walk
Walker Run
Normalized Score
100
Norallzzdd lized Score
75
50
25
200
400
200
400
Env. Steps (1e3)
Env. Steps (1e3)
Dreamerv3
DreamerV3-Align
NCRL
Figure 15: Comparison of Dreamer V3 under different model size configurations. NCRL consis- tently outperforms both variants.
A.6 PERFORMANCE ON CHALLENGING METAWORLD TASKS
In Sec. I, although NCRL solves most MetaWorld tasks with satisfactory performance, a few tasks still exhibit relatively low success rates with 150k environment steps. These tasks typically involve long horizons, small objects of interest, or strict success criteria. We have already shown three of these challenging tasks in Fig. 3, showing increased success rates with larger training budget.
20

---

denotes the Published as a conference paper at ICLR 2026
We now include additional experiments other selected tasks using increased training budget
in Fig. 16. We found that, for most tasks, the success rate improves as the training budget increases.
Reach
Basketball
Pick Place
Normalized Score
100
Normalized Score
100
Normalized Score
75
50
50
50
25
200
400
Env. Steps (1e3)
Env. Steps (1e3)
Env. Steps (1e3)
NCRL
NCRL (150K)
Figure 16: Improved success rate on Meta World tasks the training budget increases.
B THEORETICAL ANALYSIS
In this section, we give theoretical analysis of the main conclusions in our paper.
B.1 PROOF OF THE BENEFITS OF EXPERIENCE RETRIEVAL.
Proposition 1. Experience retrieval reduces distribution shift during online fine-tuning, compared to using the full offline dataset directly, in the sense that
E Bxr-prnteed: 8ax ro [||s Som|12) < Espepig, Sart-po [||s
Son
(6)
Proof. Let Par(s), (s), (s) and Pretrieved (8) denote the state distributions of the non-curated offline dataset Doff, the online dataset Don and the retrieved dataset 'D Dretrieved C Daffs respectively. We simplify the notation
Egnp, sum-aa [||s Son [a]
Espp [d(s, Son)]
Since Dretrieved C Datt the distribution (on(s) can expressed mixture distribution: Por(s) (retriewed (s) + (1 a) (s),
where (s) is the distribution over the remaining offline data, and a = Derioeal Dail
fraction of samples in the retrieved dataset.
The expected total variation for the mixture distribution decomposes as:
Eanpa [((s, Sea) =
Egr-paroas [d(s, Son)] )1 + a) Egrppe [d(s Son)]
Assume that D retiooedd is constructed by selecting states such that Sieuiieeedd Sea for some small e > O. Consequently, states Drea satisfy ||rest - Son! 2 E. This construction implies the following bounds:
Eg Egrepaninnl
[d(s, Sun) < <
(8)
E Egxppin (d(s, Sun) 2
(9)
for some > 0. Therefore, it follows that
Ex-paa [d(s, Son)
> E, Spariintnn (d(s,8ml)
Substituting into Equation equation yields:
Egg-pat [d(s, 8um)]
> Passeieseeed (d(s, Soa) (d(s, Sun)
(V E.g- Preceeedd
+ (1 - -1) EEa-pp [d(s, san)] + (1 a) Ex--eesiees [d( San)
E Eerepaannn [d(s, son]].
Thus, the expected total variation between the retrieved data and online data is strictly smaller than that between the full offline data and online data.
21

---

Published as a conference paper at ICLR 2026
Explanation. Experience retrieval helps prevent catastrophic forgetting during online fine-tuning.
Definition B.1 (Catastrophic Forgetting due to Data Distribution Shift). Catastrophic forgetting occurs when a neural network, after training on a new data distribution, experiences a significant performance drop on previously learned tasks due to the overwriting of representations from earlier distributions, caused by biased parameter updates towards the new distribution.
Proof. Following the previous notations, let Don and Deerrived denote the online dataset and the retrieved offline dataset, respectively. The objective in Eq. (1) can be written as:
Emixed (0) = (o) + Cretrieved (0)
Epa.go (o,a)~DD
In 22, he) + B KL (go(z) h2, or) |l po(zs [ha))
A- Epa.dee de (o.0)= Dranised
In (o ze, h1) + B KL (g0 (e, |l po(z ho))
Assuming the À is monotonic function of a Pranterr
and À > O, since Dretrivvet C Dor, the term
Das
freiiieee (0) acts as a regularizer during online updates, constraining parameter changes on Dan in way that preserves performance on the retrieved offline distribution Pretrieved This mitigates the risk of catastrophic forgetting by anchoring the model to previously seen data.
B.2 PROOF OF IMPROVED PERFORMANCE WITH EXECUTION GUIDANCE
Proposition 2 (Performance Improvement via Execution Guidance). Let nt denote exploration policy and T guide policy obtained via imitation learning. Let e = max, (-1)) (8, a)ll
Let be mixed policy (execution guidance) derived from n² and n9, defined as: # *(a) s) = (al()))) + - (als), E (0.
(10)
Then, the performance of the mixed policy स exceeds that of the exploration policy x² by at least:
n) - ((T)) 2
Egndy
(11)
1
1
-20E
1-
1-7(1-a)
where y E [o, 1) is the discount factor.
Proof. The proof follows directly from Theorem 4.1 Kakade & Langford (2002): we just need to replace n' in Kakade & Langford (2002) with T and in Kakade & Langford (2002) with л². According to (Kakade & Langford, 2002) an example can be provided where this bound is tight.
This establishes that the performance improvement of the mixed policy म over the exploration policy Te is positive when the expected policy improvement of the guidance policy over the execution policy Exrdge IE,, x² (-)s))A, (St, a)] is larger than the term -2(1 - -)e((- 1-x(1-a) which results from the distribution shift due to using the guidance policy instead of only the execution policy.
Moreover, according to Corallary 4.2 in Kakade & Langford (2002), when Egsdd_ IΣ,- T² (-Is)A,;= a)] 2 and when the maximal immediate reward is positive, we can always choose such that the performance improvement is positive (see Kakade & Langford (2002) for details).
22

---

Published as a conference paper at ICLR 2026
C MORE RELATED WORK
In this section, we give a more detailed related work review.
RL with task-specific offline datasets Leveraging offline data is a promising direction to improve sample efficiency in RL. One representative approach is offline RL, which trains agents using of- fline data without environment interaction. These methods typically constrain the distance between learned and behavior policies in different ways (Kumar et al., 2020; Fujimoto & Gu, 2021; Kumar et al., 2019; Wu et al., 2019; Kostrikov et al., 2021; 2022; Uchendu et al., 2023). However, policy performance is highly dependent on dataset quality (Yarats et al., 2022). To enable continued im- provement, offline-to-online RL methods (Leee et al., 2022; Zhao al., 2022; Yu & Zhang, 2023; Nair et al., 2020; Rafailov et al., 2023) were developed, which fine-tune policies trained with of- fline RL by interacting with environments. MOTO (Rafailov et al., 2023) proposes model-based RL method with reward-labeled data, and requires model-based value expansion, policy regularization, and controlling epistemic uncertainty, while our method leverages reward-free and malti-embodiimet data and requires none of the techniques proposed by MOTO.
Typical offline-to dffline-to-onligg to-online RL face training instability challenges (Lee et al., 2022: Lu et al., 2023). To mitigate this issue, RLPD (Ball et al., 2023) is proposed and demonstrates strong performance by simply concatenating offline and online data, but requires reward-labeled task-specific offline data and does not address multi-embodiment scenarios. ExPLORe (Li et al., 2023) labels reward-free offline data using approximated upper confidence bounds (UCB) to solve hard exploration tasks, but relies on near-expert data for the target tasks, while we consider a more general setting with non-curated data.
RL with multi-task offline datasets Recent work has explored multi-task offline RL (Kumar et al., 2023; Hansen et al., 2024; Julian et al., 2020; Kalashnikov et al., 2021; Yu et al., 2021), but requires known rewards. PWM (Georgiev et al., 2024) and TDMPC-v2 (Hansen et al., 2024) train world models for multi-task RI but are limited to state-based inputs and reward-laheled data. To handle unknown rewards, approaches like human labeling (Cabi al., 2020; Singh et al., 2019), inverse RL (Ng et al.. 2000; Abbeel & Ng, 2004), or generative adversarial imitation learning (Ho & Ermon, 2016) can be used, though these require human labor or expert demonstrations. Yu al. (2022) assigns zero rewards to unlabeled data, which introduces additional bias. Apart from these, there is a line of work that focuses on representation learning from in-the-wild data (Schwarzer et al., 2021; Parisi et al., 2022; Yang & Nachum, 2021; Yuan et al., 2022; Stooke et al., 2021; Shah & Kumar, 2021; Wang et al., 2022; Sun et al., 2023; Ze et al., 2023; Ghosh et al., 2023) but fails to utilize rich information in the dataset, such as dynamics.
Recent studies (Sco et al., 2022; Wu et al., 2025; 2023) explore world model pre-training with action- free data, focusing on world model architecture design to utilize the action-free data. However, we demonstrate that naive fine-tuning of pre- pre-trained world models fails on challenging tasks, while method, incorporating experience rehearsal and execution guidance, significantly improves RL performance across 72 tasks.
Unsupervised RL In unsupervised RI., an agent explores the environment based intrinsic mo- tivations, and the models' parameters are initialized during this self-motivated exploration stage, aiming for fast downstream task learning (Rajeswar al., 2023; Pathak et al., 2017; Burda et al., 2019; Eysenbach et al., 2019; Pathak et al., 2019; Liu & Abbeel, 202 Sekar et al., 2020; Liu & Abbeel, 2021b; Yarats et al., 2021: Laskin al., 2021; Mazzaglia et al., 2023; Xu al., 2022). Our problem setting differs from unsupervised RL in several ways: i) Unsupervised RL interacts with the environment actively while we leverage static offline datasets, ii) unsupervised RL gives spe- cific focus on designing different intrinsic rewards, while our setting focuses on improving sample efficiency by leveraging unlabeled datasets.
Generalist Agents RL methods usually perform well on a single task (Vinyals et al., 2019; Andrychowicz et al., 2020), however, this contrasts with humans who can perform multiple tasks well. Recent works have proposed generalist agents that master diverse set of tasks with a single agent (Reed et al., 2022; Brohan et al., 2023b; Team et al., 2024; Zhao et al., 2024). These methods typically resort to scalable models and large datasets and are trained via imitation learning (Brohan et al., 2023b;a; O' Neilll et al., 2023; Khazatsky et al., 2024). In contrast, we train a agon-iio world model and use it to boost RL performance for multiple tasks and embodiments.
23

---

Published as a conference paper at ICLR 2026
World models World models learn to predict future observations states based historical information. World models have been widely investigated in online model-based RL (Hafner et al., 2020: Ha & Schmidhuber, 2018; Micheli et al., 2023; Alonso et al., 2024; Scannell et al., 2025). Recently, the community has started investigating scaling world models (Ha & Schmidhuher, 2018), for example, Hu al. (2023a); Pearce et al. (2024); Wu al. (2025); Agarwal et al. (2025); Mereu et al. (2025) train world models with Diffusion Models or Transformers. However, these models are usually trained on demonstration data. In contrast, we explore the offlnneto-online RL setting closely fitting the pre-train and then fine-tune paradigm and we focus on leveraging reward-free and multi-embodiment data to increase the amount of available data for pre-traine pre-training. We further identify the distributional shift issue when fine-tuning the pre-trained world model and mitigate the issue by proposing experience rehearsal and execution guidance.
D LIMITATIONS
Although demonstrating strong performance on a diverse set tasks, our method has the following limitations. 1) The world model architecture used in paper is the recurrent state space model. This model built upon RNN, which can be limited for scaling. This can be mitigated by using Transformer and liffusion-basee world model. However, we note that the main conclusion of this paper should still be valid. 2) We do not thoroughly discuss the generalization ability of the pre-trained world model. With DMControl tasks, our method shows a promising trend in general- izing to unseen tasks. However, generalization to new embodiments or novel configurations is still challenging. which requires even diverse training data. 3) The non-curated offline data used in our paper, although lifting several key assumptions in previous difline-to-onlin data, i.c., our current method not able leverage the vast in-the-wild data. A promising direc- tion is to combine in-the-wil data for pre-training as in (Wu et al., 2025) and the domain- specific "in-house" data (as used in our paper) for post-training. 4) We only conduct experiments in the sim- ulator. Considering the sample efficiency of our proposed method, it could be promising to conduct experiments on real-world applications.
RL, is still in- in-domain
E DISCLOSURE OF LLMS USAGE
Large Language Models (LLMs) were used to assist word choice, improving grammar as well as proof checking in Sec. B. LLMs were also used in compressing the Related Work section due to page limits. The Related Work section was initially written by authors without using LLMs and the compressed text was subsequently revised by the authors. The main draft was written by authors without using LLMs. The ideas were formalized independently of LLMs assistance.
F COMPUTE RESOURCES
We conduct all experiments on clusters equipped with AMD MI250X GPUs, 64-core AMD EPYC "Trento" CPUs, and 64 GBs DDR4 memory. For pre-training. takes ~ 48 GPU hours for 150k steps. For fine-tuning, it tasks ~ 8 GPU hours per run for 150K environment steps. Note that due to AMD GPUs not supporting hardware rendering, the training time should be longer than using Nvidia GPUs. To reproduce the NCRL's results in Fig. 3, it roughly takes 8 h * 72 tasks * 3 seeds = 1728 GPU hours.
G IMPLEMENTATION DETAILS
G.1 BEHAVIOR CLONING
The Behavior Cloning methods used in both the execution guidance of NCRL and JSRL-BC are the same. We use a four-layer convolutional neural network (LeCun et al., 1995) with kernel depth [32, 64, 128, 256] following three-layer MLPs with LayerNorm (Ba et al., 2016) after all linear layers.
We list the adopted encoder and actor architectures for reference.
24

---

Published as a conference paper at ICLR 2026
class Encoder (nn. :
(self, obs_shape);
super () (ono)())niii.__.. iniit(()
assert obs_shape = (9, 64, 64), f' obs_shappe is { (obs_shapp) }, but expect (9, 64, 64) inputs shape
relf.repr_de dim (32 8) 2 _inpt__chnnel 9
self.convnet Sequential
10
an.Conv2d i_npddd___nntttcnane_.. 32, 4, stride-2), [8, 32, 31, 31] nn.Elu(),
11
12
nn. (32, 32+2, 4, stride-2), F[B, 64, 14, 14] nn.ELU(),
Cn.von (32+2, 32*4, 4, stride=2), I 128, 6, 6] nn.ELU(),
15
16
Cn.oonvdd(32+: 32+8, 4, stride=2), [BB 256, 2, 2] nn.ELU())
17
self.apply
init)
20
def forward (self, obs) B, c, H, W obs. shape
21
22
obs obs 255.0 0.5
24
convnet (ops) I reshape to [B, -1]
25
26
h h.view (B, -1)
return h
class Actor (nn. Module):
def wiinitt inntt (self, repr_dim, acticn_shape, feature_dim-50, hi hidden_din
-1024)
super () init_
self.trunk nn. Sequential (flayerrormmffeturee..
www.Liieerrreer__dde feature_dim),
Tanh ())
self.policy = nn. Sequentiai
(nn. Linear Linea (feature dim, hidden bidden_dim),
nn. Layor (hidden_dim)) nn an.ELU()
nn Linear (hidden dim, hidden_dim), nn nn. La. (hi nn.ELU
10
11
nn. Linear (hidden_die dim, (stion_shape(0))
12
self. self.apply (uti fotill weacclls.weightiiiiie
15
def forward obs, std)
16
= self.trume (obs)
17
return polf.policy (h)
G.2 JSRL+BC
Jump-start RI. (Uchendu et al., 2023) is proposed as an offline-to-online RL method. includes two policies, prior policy ne, (als) and a behavior policy To, (a]s), where the prior policy is trained via offline RL methods and the behavior policy is updated during the online learning stage. However, offline RL requires the offline dataset to include rewards for the target task. To extract behavior policy from the offline dataset, we use the BC agent described above as the prior policy. During online training, in each episode, a pre-defined array np- ar aragge ( (0, 101, 10). We then execute the prior policy for h steps and switch to the behavior policy until the end of episode.
randomly sample the rollout horizon h of the prior policy from
25

---

Published as a conference paper at ICLR 2026
G.3 EXPLORE
For the ExPLORe baseline, we follow the original training code?. We sweep over several design choices: i) kernel size of the linear layer used in the RND and reward models: [256 (default), 512]; ii) initial temperature value: [0.1 (default), 1.0]; iii) whether to use LayerNorm Layer (no by default); iv) learning rate: [1c-4, 3e-4 (default)] However, we fail to obtain satisfactory performance. There are several potential reasons: i) the parameters used in the ExPLORe paper are tuned specifically to their setting, where manipulation tasks and near-expert trajectories are used; ii) the coefficient term of the RND value needs to be tuned carefully for different tasks and the reward should also be properly normalized.
To achieve reasonable performance and eliminate the performance gap caused by implementation- level details, we make the following modifications: we replace the RND module with ensembles to calculate uncertainty; ii) the reward function shares the latent space with the actor and critic.
Source code of ExPLORe http://iitbcccm/ccccchrcrree////
26

---

Published as a conference paper at ICLR 2026
H ALGORITHM
The full algorithm is described in Alg. 1.
Algorithm Efficient RI. by Guiding World Models with Non-Curated Offline Data
Require:
Non-curated offline data Dotr, Online data Don 0, Retrieval data Dretrieval World model fe, qo Pa: dg Policy Tom. Touc. Value function us and Reward rg.
Task-Agnossic World Model Pre-Training for num. pre-train steps do
Randomly sample mini-batch Bott {Or, at Or4 J from Dofr- end for
Update world model fe, qo, pa, da by minimizing Eq. (1) on sampled batch B.
Task-Span Specifiii Training
// Experience Retrieval
Collect one initial observation og from the environment.
Compute the visual similarity between Oon and initial observations of trajectories Coff in Dott us-
ing Eq. (5).
Select trajectories according to Eq. (5) and fill D retrieaal-
Behavior Cloning Policy Training for num. be updates do
Randomly sample mini-batch Breuioval for, at Ji from Deetrivall-
Update Téec by minimizing - > log ग ducc (0rlor).
end for
Task- Pask-Specific RL Fine-Tuning for num. episodes do
Collect Data
Decide whether to use dduc according to the predefined schedule.
Select π The then
Randomly select the starting time step ic and the rollout horizon H.
end
I
while 1 episode length do
0t Téac (azlo;) Use Téne and k < t H else a, TòRi (1)) Interact with the environment using Store (or, rz, o to D
f 1+11
end while
// Update Models
num. grad steps do Randomly sample mini-batch Bon for, a,
Ti, T ott1 from Don and Beerivaal
for, Oe, Tt, 1 from Drenioval-
Update world model fo, 9o Po. dg by minimizing Eq. (1) sampled batch {B... Brerieval } Update rş by minimizing N Ein log (re(s))) on B.n-
8: [he, 21
// Update policy and value function
Generate imaginary trajectories (0r, az, 814 1 by rolling out ha, po with Toa Update policy Tom. and value function us with Eq. (4).
end for
end for
27

---

Published as a conference paper at ICLR 2026
I FULL RESULTS
In Table and Table 4, we list the success rate of 50 Meta- World benchmark tasks with pixel inputs. In Table 5, list the episodic return of DMControl o2 tasks. We compare NCRL at 150k samples with two widely used baselines Dreamer V3 and DrQ-v2 at both 150k samples and 1M samples. We report the results over random seeds for NCRL and 3 random seeds for DreamerV3 and DrQ-v2. The best results are marked with a bold font at 150k samples and the highest overall scores are marked with underline. The detailed result curves of both Meta-World World and DMControl are shown in Fig. 17, Fig. 18, and Fig. 19.
1.1 META-WORLD BENCHMARK
Table 3: Success rate of Meta-World benchmark with pixel inputs.
DreamerV3 1M
DrQ-v2 1M
DreamerV3 @ 150k
DrQ-v2 150k
NCRL
Tasks
150k
Assembly
0.0
0.0
0.0
0.0
0.44
Basketball
0.0
0.97
0.0
0.0
0.36
Bin Picking
0.0
0.93
0.0
0.33
0.84
Box Close
0.13
0.9
0.0
0.0
0.88
Button Press
0.7
0.47
0.76
Button Press
1.0
1.0
0.33
0.17
1.0
Topdown
Button Press
1.0
1.0
0.73
0.63
1.0
Topdown Wall
Button Press Wall
1.0
1.0
0.93
0.77
1.0
Coffee Button
1.0
1.0
1.0
1.0
1.0
Coffee Pull
0.6
0.8
0.0
0.6
0.56
Coffee Push
0.67
0.77
0.13
0.2
0.72
Dial Turn
0.67
0.43
0.13
0.17
0.65
Disassemble
0.0
0.0
0.0
0.0
0.0
Door Close
1.0
Door Lock
1.0
0.93
0.97
0.96
Door Open
1.0
0.97
0.0
0.0
0.92
Door Unlock
1.0
1.0
1.0
0.63
0.92
Drawer Close
0.93
1.0
0.93
1.0
0.92
Drawer Open
0.67
0.33
0.13
0.33
1.0
Faucet Open
1.0
1.0
0.47
0.33
1.0
Faucet Close
0.87
1.0
1.0
0.92
Hammer
1.0
1.0
0.07
0.4
1.0
Hand Insert
0.07
0.57
0.0
0.1
0.44
Handle Press Side
1.0
1.0
1.0
1.0
1.0
Handle Press
1.0
1.0
0.93
0.97
1.0
Handle Pull Side
0.67
1.0
0.67
1.0
Handle Pull
0.67
0.33
0.6
0.85
Lever Pull
0.73
0.83
0.0
0.33
0.72
More results see Table 4
28

---

Published as a conference paper at ICLR 2026
Table 4: Success rate of Meta- World benchmark with pixel inputs (Cont.).
DreamerV3 @ IM
DrQ-v2 @ 1M
DreamerV3 @ 150k
DrQ-v2 @ 150k
NCRL (ours)
Tasks
150k
Peg Insert Side
1.0
1.0
0.0
0.27
1.0
Peg Unplug Side
0.93
0.9
0.53
0.5
0.48
Pick Out of Hole
0.0
0.27
0.0
0.0
0.25
Pick Place Wall
0.2
0.17
0.0
0.0
0.64
Pick Place
0.67
0.67
0.0
0.0
0.20
Plate Slide
1.0
1.0
0.93
1.0
1.0
Back Side
Plate Slide Back
1.0
1.0
0.8
0.97
1.0
Plate Slide Side
0.9
0.73
0.5
0.52
Plate Slide
1.0
1.0
0.93
1.0
0.95
Push Back
0.33
0.33
0.0
0.0
0.32
Push Wall
0.33
0.57
0.0
0.0
0.84
Push
0.26
0.93
0.0
0.13
0.72
Reach
0.87
0.73
0.67
0.43
0.40
Reach Wall
0.87
0.53
0.7
0.80
Shelf Place
0.4
0.43
0.0
0.0
0.80
Soccer
0.6
0.3
0.13
0.13
0.16
Stick Push
0.0
0.07
0.0
0.0
0.64
Stick Pull
0.0
0.33
0.0
0.0
0.52
Sweep Into
0.87
1.0
0.0
0.87
0.72
Sweep
0.73
0.0
0.3
0.64
Window Close
1.0
1.0
0.93
1.0
1.0
Window Open
1.0
0.97
0.6
1.0
0.96
Mean
0.656
0.753
0.360
0,430
0.748
Medium
0.870
0.900
0.130
0,330
0.840
29

---

Published as a conference paper at ICLR 2026
Assemblly
Basketball
Bin Picking
Box Close
300
Button Press
Button Press Tapdoun
Button Press Topdown Wall
Button Press Wall
Ken.
Fex.
Fex Sxp-CCe
Fea Nepp.Cte
coffee Button
Coffee Pull
Cuffee Push
Dial Tum
(iip:-12333 (242)
Dicascamble
Door Close
Dear
Door Open
100
Kea Sept.Ce
Door Unlock
Drawer Close
Drawer Open
Faucett Close
1.00
Sugu
Sapt Hand Insert
Paucot Open
Hommar
Handle Press
100
Noreetee
NCRL
Dreamerv3
DeQ 92
Figure 17: Meta-War World results, We report 5 seeds for NCRL and seeds for DrQ-v2 and Dream- erV3.
30

---

Published as a conference paper at ICLR 2026
Handie Press Side
Handle Pull
Handle Pull Side
Lever Pull
238
102
25
200
IPag Insertt Side
Pog Unplag Side
Pick Out Hole
Pick Place
Horme Bost Scme
138
112
100
506
150
Plate Sidde
Plate Slide
Sare
26
Push
Puth
235
154
75
15
30
15
50
306
125
150
25
Sweep
1401
NCRL
Dresmerka
DrO
Figure 18: Meta-World results (Cont.). We report 5 seeds for NCRL and 3 seeds for DrQ-v2 and DreamerV3.
31

---

Published as a conference paper at ICLR 2026
1.2
DMCONTROL BENCHMARK
Table 5: Episodic return of DMControl benchmark with pixel inputs.
DreamerV3 @ 500k
DrQ-v2
DreamerV3
DrQ-v2
NCRL((urs)
Tasks
@ 500k
@ 150k
@ 150k
@ 150k
CartPole Balance
994.3
992.3
955.8
983.3
995.0
Acrobot Swingup
222.1
30.3
85.2
20.8
84.6
Acrobot Swingup
2.5
1.17
1.7
1.5
12.2
Sparse
Acrobot Swingup
-0.2
0.3
2.0
0.4
-17.1
Hard
Walker Stand
965.7
947.6
946.2
742.9
974.1
Walker Walk
949.2
797.8
808.9
280.1
960.5
Walker Run
616.6
299.3
224.4
143.0
707.7
Walker Backflip
293.6
96.7
128.2
91.7
266.1
Walker Walk
942.9
744.3
625.9
470.9
887.6
Backward
Walker Walk
-2.1
-9.5
4.7
-17.1
842.8
Hard
Walker Run
363.8
246.0
229.4
167.4
366.0
Backward
Cheetah Run
843.7
338.1
621.4
251.2
543.8
Cheetah Run Front
473.8
202.4
143.1
108.4
317.6
Cheetah Run Back
657.4
294.4
407.6
171.2
462.3
Cheetah Run Backwards
693,8
384.3
626.6
335.6
521.6
Cheetah Jump
597.0
535.6
200.8
251.8
614.2
Quadruped Walk
369.3
258.
145.2
76.5
855.6
Quadruped Stand
746.0
442.2
227.2
318.9
941.4
Quadruped Run
328.1
296.5
183.0
102.8
766.9
Quadruped Jump
689.6
478.3
168.3
190.5
820.2
Quadruped Roll
663.9
446.0
207.9
126.2
948.0
Quadruped Roll
508.8
366.9
124.8
164.7
758.9
Fast
Mean
541.81
372.23
320.86
226.49
617.73
Medium
606.8
318.70
204.35
166.05
733.3
32

---

Published as a conference paper at ICLR 2026
03.
Scae
UR
12:
Sapt 240
2ept
frne Scom
U2
120
Quadruped
1.0
5
3430
240
Dr Sepr
12
15
15
U2
340
343)
340
DrQ
Figure 19: DMControl results. We report seeds for NCRL and 3 seeds for DrQ-v2 and DreamerV3,
33

---

Published as a conference paper at ICLR 2026
J HYPERPARAMETERS
In this section, we list important hyperparameters used in NCRL.
Table 6: Hyperparameters used in NCRL.
Hyperparameter
Value
Pre-training
Stacked images
Pretrain steps
200,000
Batch size
16
Sequence length
64
Replay buffer capacity
Unlimited Uniform
Replay sampling strategy
RSSM
Hidden dimension
12288
Deterministic dimension
1536
Stochastic dimension
32 96
Block number
8
Layer Norm
True
CNN channels
[96, 192, 384, 768)
Activation function
SiLU
Optimizer
Optimizer
Adam
Learning rate
le-4
Weight decay
le-6
Eps
le-5
Gradient clip
100
Fine-tuning
Warm-up frames
15000
Execution Guidance Schedule
linear( 1,0,50000) for DMControl
1inear(1,0,1,150000 for Meta- Woold
Action repeat
2
Offline data mix ratio
0.25
Discount
0.99
Discount lambda
0.95
MLPs
[512, 512, 512]
MI.ss activation
SiLU
Actor critic learning rate
8e-5
Actor entropy coef
le-4
Target critic update fraction
0.02
Imagine horizon
16
34

---

Published as a conference paper at ICLR 2026
K
TASK VISUALIZATION
Figure 20: Visualization of tasks from DMControl and Meta- World used in our paper.
35


Reviews:
[{"review_id": "B7EupRYJ2C", "reviewer": "Reviewer_RREe", "summary": "The work addresses the challenge of improving sample efficiency in reinforcement learning (RL) by leveraging non-curated offline datasets which are reward-free, of mixed quality, and collected from multiple embodiments during online reinforcement learning. They investigate this in the world modeling case, and observe that naively replaying this data fails because of distribution mismatch. To overcome this, the paper proposes NCRL (Non-Curated offline data for RL), a two-stage framework: first train a task-agnostic world model on the offline data, and then fine-tune it on the target task. Specifically, the authors introduce two techniques during the fine-tuning phase to take advantage of this data: (1) Experience Rehearsal, which retrieves task-relevant trajectories from the offline dataset (based on neural feature distance using the world model encoder) and replays it during fine-tuning, and (2) Execution Guidance via Prior Actors, which randomly alternates between a prior policy trained on the retrieved offline data and the online RL policy, improving exploration.\n\nExperiments demonstrate that in the low-sample regime, NCRL improves significantly over learning-from-scratch baselines and match their performance in the high-sample regime; it also help sin a continual learning scenario with a sequence of tasks. Careful ablation experiments show each component's contribution to the observed improvement and improvement over directly using uncertainty-aware rewards for the offline data.", "questions": "This line of work [1], [2] are relevant given the emphasis on using demonstrations in a world-modeling approach.\n\nFurther analysis of exactly how the offline dataset matches the current task distribution would be helpful, i.e. could we quantify or visualize how much of the improvement comes from retrieving trajectories matching the current task state-space? Does the fact that the dataset is cross-embodiment matter if all the retrieved trajectories are the same embodiment?\n\nAny further analysis on the ablations in Figure 6? Specifically, it seems that execution guidance is the main differentiator for achieving nonzero reward for some subset of the Meta-World tasks.\n\n[1] Hansen, Nicklas, et al. \"Modem: Accelerating visual model-based reinforcement learning with demonstrations.\" arXiv preprint arX\n[2] Lancaster, Patrick, et al. \"Modem-v2: Visuo-motor world models for real-world robot manipulation.\" 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is written with clarity; the exposition of methods and step-by-step motivation is reasonable / comprehensive. In particular, the high-level motivation for the setting is very clear.\n\nThe experimental methodology is thorough and high-quality: covers a wide range of environments, uses appropriate metrics (normalized scores, learning curves with confidence intervals), and includes ablations to justify the algorithm’s design.\n\nThe novelty is primarily in framing the setting of a mixed-quality offline dataset; the methods fall quite naturally from this, and has related work with different assumptions (e.g. JSRL).", "weaknesses": "While the high-level motivation of using mixed-quality offline dataset makes perfect sense, the details of the way the offline dataset is constructed is somewhat artificial (e.g. agents corrupted with noise) and has characteristics which naturally lends itself to the proposed method; specifically, it's likely to cover similar state-space as some of the tasks, which means smart retrieval methods from this dataset would naturally help in the low-data regime. Similarly, a prior policy trained on retrieved tasks would help exploration simply by virtue of having seen similar tasks before. Perhaps putting stronger explanation on why this kind of dataset is reasonable to assume / arises naturally (e.g. in real-world robotics settings) would better motivate the method.\n\nThe impact of showing improvements in the low-data regime with these particular simulation settings is less clear, given cheapness of simulation data and the asymptote to the same performance with more environment steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vbB6J4U65x", "reviewer": "Reviewer_LFwp", "summary": "The paper introduces NCRL (Non-Curated offline data for efficient RL), an algorithm that leverages non-curated, reward-free, mixed-quality, and multi-embodiment datasets to improve sample efficiency in offline-to-online reinforcement learning. NCRL first pretrains a multi-task, multi-embodiment world model on this data, then reuses the same offline trajectories during fine-tuning through two mechanisms:\n(1) experience rehearsal, which retrieves task-relevant trajectories to mitigate distribution shift, and\n(2) execution guidance, a behavior-cloned prior policy that intermittently steers the agent during online training.\nIn  Meta-World and DMControl tasks, NCRL outperforms baselines and matches the performance of models trained with 3.3–6.7× larger budgets. It also surpasses several offline-data and world-model pretraining baselines.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Clarity and motivation.** The paper is well-written. The contributions are clearly articulated, the relationship to prior work is carefully established, and the distinctions from existing approaches are explicit and well-justified. I especially appreciate that the authors explain the reasoning behind key design choices throughout. Figure 2 provides a particularly clear and compelling motivation for the proposed approach.\n1. **Experimental design** The experiments are thoughtfully designed and demonstrate consistent, meaningful improvements of NCRL over relevant baselines. The ablations are comprehensive and provide useful information (e.g. showing that both core components of NCRL contribute to performance gains)\n1. **Reproducibility.** The paper includes extensive implementation and hyperparameter details, making the method straightforward to reproduce and encouraging further study or application.", "weaknesses": "I vote to accept, though I have a few minor comments and clarifying questions:\n\n1. At the end of Section 3, the paper refers to theoretical results in Appendix B, but the main text does not summarize what these results show or why they are relevant. It would help readers if the paper briefly explained, in the main body, what guarantees or insights the theory provides (e.g., how experience rehearsal or execution guidance theoretically improves performance or mitigates distribution shift), etc.\n1.  In Tables 3 and 4, NCRL solve most Meta-World tasks with high probability but not all of them. Could the authors clarify what makes these tasks challenging? Are these differences due to the inherent difficulty of the tasks, the limited training budget of 150k steps, or particular weaknesses in retrieval or guidance?\n\nTypos:\n1. “reuses the this data” --> “reuses this data.”\n2. “Comparsion” --> “Comparison” in Fig. 11 caption\n4. \"reutrn\" --> \"return\" in code snippet", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rZ7s3bBeTe", "reviewer": "Reviewer_Uq2c", "summary": "This paper investigates the question of how to improve sample efficiency of RL by using offline data. The specific situation this paper looks into is the case where there is a large amount of offline data but that data is unstructured and not annotated. The paper pre-trains a world model on such data, consisting of many embodiments and then uses fine tuning to optimize for a specific task. The novel methodology of this work is the enabling of fine tuning of the world model to optimize for specific tasks and improve performance. In the first stage they pre-training their world model on offline image-based data. In the fine tuning stage their agent interacts with the environment and collects data for world model training. Their results show that their pre-training is able to improve sample efficiency greatly and doesn't hurt final performance.", "questions": "Around line 266 you describe using a behavior cloned policy to choose actions. Does this cause any issues with the data now being off policy? I'm not sure how the dreamer architecture handles off policy data (it might handle it completely fine) but I'm wondering if this was something you considered?\n\nThe central problem you identify is distribution shift between the pre train data and the fine tune data. I see why this could cause problems but would your method prevent or slow down (in some settings) convergence of the world model to the best points in the world space? I'm just thinking that keeping around old data is fine and prevents a distribution shift but what about in the cases where the distribution shift is actually required. Possible example would be long horizon tasks where we only have pre train data for a small portion of the beginning, in this case we might not learn as well the longer ends of the task.\n\nFor the results (280) did you increase the size of the dreamer world model as well? It is unclear to me if this would make any difference or if it might even make dreamer perform worse but I missed if you described this somewhere.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Great motivating question as well as a practical method.\n\nVery strong results using a single world model for multiple tasks. This is a difficult challenge and this paper provides impressive empirical results.\n\nPaper is well written and very clear. Method is described well and the paper is easy to read. Empirical results are displayed clearly.\n\nEmpirical results are impressive and seem to be statistically well done with confidence intervals and such.", "weaknesses": "I do not have any strong weaknesses in this paper that I can find.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iyCnaE52UJ", "reviewer": "Reviewer_dmPf", "summary": "This paper proposes a two-stage RL from offline data methods, which can leverage mixed quality offline data without reward labels. A world model is trained on these unlabeled offline data at stage one. At stage two, offline data relevant to online tasks, measured by the neural feature distance between online samples and offline trajectories, is retrieved to enrich the initial state distribution sampling. A behavior cloning policy $\\pi_\\text{bc}$ is pretrained on offline data to alternate sampling with the target policy during online training for enriching exploration.\n\nComprehensive experiments on locovisual tasks were conducted to verify the empirical advantages of this approach, compared to baseline RL methods that use offline data, as well as RL methods that learn from scratch.", "questions": "- Has the scaling world model to 280M parameters also been applied to baseline methods for fair comparison?\n- How does Figure 2 support this: \"The world model’s accuracy degrades on states visited by the early-stage policy, especially when the offline data distribution is narrow, hurting sample efficiency.\" (Line 213-214)\n- Does training a bc policy on offline data rely on an unmentioned prerequisite: the mix quality data is not poor data? As mentioned, \"offline data often contains valuable information like near-expert trajectories and diverse state-action coverage\" (line 260). In my intuition, if the offline data quality is low, leveraging a bc policy would not help much in this algorithm. Have the authors explored to what extent the quantity and quality of offline data impact the algorithm's performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Pros:\n\n\\+ This paper tackles an intriguing question of how to utilize imperfect, unlabeled, yet largely available offline data to boost RL performance, which is of practical importance.\n\n\\+ Technically solid methods that were clearly presented.\n\n\\+ Strong empirical performance with a gain of large margins, compared with baseline methods.\n\n\\+ Comprehensive evaluations on a wide range of baseline methods and locomotion tasks", "weaknesses": "Cons:\n\n\\- Vague descriptions about related work, such as \"uses a series of complex techniques to stabilize training\" (line 129) or \"or carefully curated dataset\" (line 131) - authors could've briefly summarized what techniques were used, and how curated data is generated in these related works.\n\n\\- Unclear definition of mixed quality data. Figure 9 alone does not sufficiently indicate the data quality, but only shows that imitation learning on these data leads to limited performance. See my question below for a follow-up.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)