Finetuning Offline World Models in the Real World
Authors: Yunhai Feng
Nicklas Hansen
Ziyan Xiong
Chandramouli Rajagopalan
Xiaolong Wang
Venue: CoRL 2023 Oral (2023)
Keywords: Model-Based Reinforcement Learning
Real-World Robotics

Abstract: Reinforcement Learning (RL) is notoriously data-inefficient, which makes training on a real robot difficult. While model-based RL algorithms (world models) improve data-efficiency to some extent, they still require hours or days of interaction to learn skills. Recently, offline RL has been proposed as a framework for training RL policies on pre-existing datasets without any online interaction. However, constraining an algorithm to a fixed dataset induces a state-action distribution shift between training and inference, and limits its applicability to new tasks. In this work, we seek to get the best of both worlds: we consider the problem of pretraining a world model with offline data collected on a real robot, and then finetuning the model on online data collected by planning with the learned model. To mitigate extrapolation errors during online interaction, we propose to regularize the planner at test-time by balancing estimated returns and (epistemic) model uncertainty. We evaluate our method on a variety of visuo-motor control tasks in simulation and on a real robot, and find that our method enables few-shot finetuning to seen and unseen tasks even when offline data is limited. Videos are available at https://yunhaifeng.com/FOWM

Finetuning Offline World Models in the Real World
Yunhai Feng 1-
Nicklas Hansen
Ziyan Xiong 12+
Chandramouli Rajagopalan'
Xiaolong Wang'
'University of California San Diego
Tsinghua University
Abstract: Reinforcement Learning (RL) is notoriously data-inefficient, which makes training real robot difficult. While model-based RI. algorithms (world models) improve data-efficiency some extent, they still require hours days of interaction to learn skills. Recently, offline RL has been proposed as framework for training RL policies on pre-existing datasets without any online interaction. However, constraining an algorithm to a fixed dataset induces state-action dis- tribution shift between training and inference, and limits its applicability to new tasks. In this work, we seek to get the best of both worlds: we consider the prob- lem of pretraining world model with offline data collected then finetuning the model online data collected by planning with the learned model. To mitigate extrapolation errors during online interaction, we propose regularize the planner at test-time by balancing estimated returns and (epistemic) model uncertainty. We evaluate our method on a variety of visuo-motor control tasks simulation and on a real robot, and find that our method enables few-shot finetuning to seen and unseen tasks even when offline data is limited. Videos are available https: http///ynnaaannnn.ee fer.m/offfnnccm cOm/FOWM.
real robot, and
Keywords: Model-Based Reinforcement Learning Real- Real-World Robotics
Introduction
Reinforcement Learning (RL) has the potential to train physical robots to perform complex tasks autonomously by interacting with the environment and receiving supervisory feedback in the form of rewards. However, RL. algorithms are notoriously data-inefficient and require large amounts (often millions or even billions) of online environment interactions to learn skills due to limited supervision [1. 2, 3]. This makes training on a real robot difficult. To eircumvent the issue, prior work commonly rely custom-built simulators [4, 5, 1] human teleoperation [6, for behavior learning, both of which are difficult to scale due to the enormous cost and engineering involved. Additionally, these solutions each introduce additional technical challenges such as the simulation- to-real gap [8, 1, 9, 10] and the inability to improve over human operators [11, 12], respectively.
Recently, offline RL has been proposed as framework for training RL policies from pre-existing interaction datasets without the need for online data collection [13, 14, 15, 16, 17, 18, 19, 20). Leveraging existing datasets alleviates the problem of data-inefficiency without suffering from the aforementioned limitations. However, any pre-existing dataset will invariably not cover the entire state-action space, which leads to (potentially severe) extrapolation errors, and consequently forces algorithms to learn overly conservative policies [21, 22, 23, 24]. We argue that extrapolation errors are less of an issue in an online RL setting, since the ability to collect new data provides an in- trinsic self-calibration mechanism: by executing overestimated actions and receiving (comparably) negative feedback, value estimations can be adjusted accordingly.
In this work, we seek to get the best of both worlds. We consider the problem of pretraining an RL policy on pre-existing interaction data, and subsequently finetuning said policy on limited amount
"qqual contribution. Correspondence Yunhai Feng< <yuu0200uuusd..du.>>
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Online Interaction
Q
horizon
uncerfalnily
online
balanced
sampling up
a,
offiine
buffer
Model Learning.
Planning
Figure I. Approach. We propose a framework for offline pretraining and online finetuning of world models directly in the real world, without reliance on simulators or synthetic data. Our method iteratively collects new data by planning with the learned model, and finetunes the model combination of pre-existing data and newly collected data. Our method can be finetuned few-shot unseen task variations in <00 trials by leveraging novel fest-time regularization during planning.
of data collected by online interaction. Because we consider only a very limited amount of online interactions ($20 trials), we shift our attention to model-based RL (MBRL) [25] and the TD-MPC [26] algorithm in particular, due to data-efficient learning. However, as our experiments will reveal, MBRL alone does not suffice an offline-to-online finetuning setting when data scarce. In particular, find that planning suffers from extrapolation errors when queried on unseen state- action pairs, and consequently fails to converge. While this issue is reminiscent of overestimation emror in model-free algorithms, MBRI. algorithms that leverage planning have an intriguing prop- erty: their (planning) policy is non-parametric and can optimize arbitrary objectives without any gradient updates. Motivated by this key insight, we propose framework for offline-to fofiine-to-oninee fine- tuning of MBRI. agents (world models) that mitigates extrapolation errors in planning via novel test-time behavior regularization based on (epistemic) model uncertainty. Notably, this regularizer can be applied to both purely offline world models and during finetuning.
We evaluate our method on variety of continuous control tasks in simulation, well as visuo- motor control tasks on real xArm robot. We find that our method outperforms state-of-the-arr methods for offline and online RL across most tasks, and enables few-shot finetuning to unseen tasks and task variations even when offline data limited. For example, our method improves the success rate of an offline world model from 22% to 67% in just 20 trials for real-world visual pick task with unseen distractors. We are, to the best of our knowledge, the first work investigate dffline-to-online finetuning with MBRL on real robots, and hope that our encouraging few-shot results will inspire further research in this direction.
Preliminaries: Reinforcement Learning and the TD-MPC Algorithm
We start by introducing our problem setting and MBRL algorithm choice, TD-MPC, which gether form the basis for the technical discussion of our approach in Section 3.
Reinforcement Learning We consider the problem of learning visuo-motor control policy by interaction, formalized by the standard RL framework for infinite-horizon Partially Observable Markov Decision Processes (POMDPs) [27]. Concretely, we aim to learn policy na: S A + R. that outputs a conditional probability distribution over actions E A conditioned on a state s E S that maximizes the expected return (cumulative reward) R = E,, D> rr), where t a discrete time step, Ft is the reward received by executing action a, in state S, at time t, and E [0, 1) is a discount factor. We leverage an MBRL algorithm in practice, which decomposes To into multiple learnable components (a world model), and the learned model for planning. For brevity, we subscript 0 to symbolize learnable parameters throughout this work. In POMDP, environment interactions obey (unknown) transition function T: S A + S, where states themselves are as-

---

sumed unobservable. However, we can define approximate environment states s = (01, 02 og....,,o from sensory observations 11n obtained from egg. cameras or robot proprioceptive information.
TD-MPC Our work extends TD-MPC [26], an MBRL algo- rithm that plans using Model Predictive Control (MPC) with a world model and terminal value function that is jointly learned via Temporal Difference (TD) learning. TD-MPC has two in- triguing properties that make it particularly relevant to our set- ting: (i) it uses planning, which allows us to regularize action selection at test-time, and (ii) it is lightweight relative to other MBRL algorithms, which allows us to run real-time. We sum- marize the architecture Figure Concretely, five components: (1) a representation z hg(s) that maps high-dimensional inputs a compact latent representation z, (2) latent dynamics model z' dø dø(z, a) that predicts the latent representation at the next timestep, and three prediction heads: (3) a reward predictor = R2(z, a) that predicts the in- stantaneous reward, (4) a terminal value function q = Qo a), and (5) a latent policy guide = (z) that used as behav- ioral prior for planning. We use z', denote the successor (latent) states of z, s in a subsequence, and use A, dif- ferentiate predictions from observed (ground-truth) quantities a, r. In its original formulation, TD-MPC is an online off-policy RL algorithm that maintains replay buffer B of interactions, and jointly optimizes all components by minimizing the objective
P
GH
82
30
31
22 1.e
Zo
Z1
Z2
So
S1
S2
Figure 2. Architecture. Our world model encodes an observa- tion So into its latent representa- tion Zo, and then recurrently pre- dicts future latent states Z1:h as well as optimal actions âg.hs re- wards forh, and values qo:k- Fu- ture states S1sh provide supervi- sion for learning but are not quired for planning.
Qu an) |
h
[(0)= =
E
| - (s lif - r ll - 2012
(1)
t=
Latent dynamiis
Reward
Value
Action
where as, is a subsequence of length h sampled from the replay buffer, is expo- nentially moving average of 0, sg is the stop-grad operator, dr = rz + (2 To())) is the TD-target, and gradients of the last term (action) are taken only wrt. the policy parameters. Con- stant coefficients balancing the losses are omitted. We refer to Hansen et al. [26] for additional implementation details, and instead focus our discussion on details pertaining to our contributions. During inference, TD-MPC plans actions using a sampling-
planner (MPPI) [28] that itera- tively fits a time-dependent multivariate Gaussian with diagonal covariance over the space of action sequences such that return evaluated by simulating actions with the learned model is maxi- mized. For a (latent) state Za ho(s)) and sampled action sequence ag.h. the estimated return R is given by
R == thth (o,,,, a4)
? Ro(z,, 2++1 dg (Ze; a.)
Zo
ha Encoder
Value
Reward
Latent dynamics
To improve the rate of convergence in planning, fraction of sampled action sequences are gener- ated by the learned policy тe. effectively inducing a behavioral prior over possible action sequences. While Te is implemented as a deterministic policy, Gaussian action noise can be injected for stochas- ticity. TD-MPC has demonstrated excellent data-efficiency in an online RL setting, but suffers from extrapolation errors when naively applied to our problem setting, which we discuss Section 3.
3 Approach: A Test-Time Regularized World Model and Planner
We propose a framework for offline-to-ollne finetuning of world models that mitigates extrapo- lation errors in the model via novel test-time regularization during planning. Our framework is summarized in Figure 1, and consists of two stages: (1) an offline stage where world model is pre- trained on pre-existing offline data, and (2) an online stage where the learned model is subsequently
3

---

finetuned on a limited amount of online interaction data. While we use TD-MPC [26] as our back- bone world model and planner, our approach is broadly applicable to any MBRL. algorithm that uses planning. We start by outlining the key source of model extrapolation errors when used for offline RL, then introduce our test-time regularizer, and conclude the section with additional techniques that we empirically find helpful for the few-shot finetuning of world models.
3.1 Extrapolation Errors in World Models Trained by Offline RL
All methods suffer from extrapolation errors when trained on offline data and evaluated on unseen data due to a state-action distribution shift between the two datasets. In this context, value overesti- mation in model-free Q-learning methods is the most well-understood type of error [14, 16, 17, 19]. However, MBRL algorithms like TD-MPC face unique challenges in an offline setting: state-action distribution shifts are present not only value estimation, but also in (latent) dynamics and reward prediction when estimating the return of sampled trajectories as in Equation 2. We will first address value overestimation, and then jointly address other types of extrapolation errors in Section 3.2.
Inspired by Implicit Q-learning (QQL:; [19)), we choose to mitigate the overestimation issue by ap- plying TD-backups only on in-sample actions. Specifically, consider the value term in Equation that computes TD-target q by querying Qo on a latent state z and potentially out-of-sample ac- tion Fe (z.). To avoid out-of-sample actions in the TD-target, introduce state-conditional value estimator Vo and reformulate the TD-target q: rt (V,)) This estimator be optimized by an asymmetric f-loss (expectile regression):
(0) = J - (Qo(z, a.) Vo(z.))²
(3)
where + E (0, 1) is a constant expectile. Intuitively, we approximate the maximization in (z(Z)) = maxa, (Z, a,) for t + 1, and are increasingly conservative for smaller т. Note that a, is the action from the dataset (replay buffer), and thus no out-of-sample actions are needed. For the same purpose of avoiding out-of-sample actions, we replace the action term for policy learning in Equation by an advantage weighted regression (AWR) [29, 30, 21] loss exp(B(Qo(2z. ax) Vz()))) log To (a.(Zz), where 3 2 0 is temperature parameter.
3.2 Uncertainty Estimation as Test-Time Behavior Regularization
While only applying TD-backups on in-sample actions is effective at mitigating value overestimation during offline training, the world model (including dynamics, reward predictor, and value function) may still be queried on unseen state-action pairs during planning, i.e., when estimating returns using Equation 2. This can result in severe extrapolation errors despite cautiously learned value function. To address this additional source of errors, we propose a test-time behavior regularization technique that balances estimated returns and (epistemic) model uncertainty when evaluating Equation 2 for sampled action sequences. By regularizing estimated returns, we retain the expressiveness of plan- ning with a world model despite imperfect state-action coverage. Avoiding actions for which the outcome is highly uncertain is strongly desired an offline RL setting where no additional data can be collected, but conservative policies likewise limit exploration an online RL setting. Unlike prior offline RL methods that predominantly learn an explicitly and consistently conservative value function and/or policy, regularizing planning based on model uncertainty has an intriguing property: as planning continues to cautiously explore and the model is finetuned on new data, the epistemic uncertainty naturally decreases. This makes test-time regularization based on model uncertainty highly suitable for both few-shot finetuning and continued finetuning over many trajectories.
To obtain a good proxy for epistemic model uncertainty [31] with minimal computational over- head and architectural changes, we propose utilize small ensemble of value functions Qf") Q²²² ...., Q6 similar to Chen et al. [32]. We optimize the Q-functions with TD-targets discussed in Section 3. and use a random subset of (target) Q-networks to estimate the Q-values in Equation 3. Although ensembling all components of the world model may yield a better estimate of epistemic uncertainty, choice of ensembling is lightweight enough to run real robot and is empirically sufficiently good proxy for uncertainty. We argue that with Qenssemble we can

---

Simulation tasks
Real-world tasks
Transfer tasks
Robot setup
Figure 3. Tasks. We consider diverse tasks in simulation and real robot. Our real-world tasks use raw pixels as input. Our method achieves high success rates in offline-tooonling transfer to both seen and unseen tasks in just 20 online trials on a real robot.
Table 1. Real-world offline-to-online results. Success rate (%) as a function of online finetun- ing trials. Mean of 18 trials and 2 seeds.
Table 2. Finetuning to unseen real-world tasks. Success rate (%) of our method for each task variation shown in Figure 3. We include 4 successful transfers and failure. See Ap- pendix B for task descriptions. Mean of 18 trials and seeds.
online
offline-to-online
Trials
TD-MPC
TD-MPC
Ours
Reach
0
Oso
50218
72:1
10
Oso
67
94:6
online trials
20
O+o
78=12
89:
Variation
10
20
Pick
0
O4o
0.:0
01:0
Reach
distractor
22:0
22:11
62:16
10
O.
28
3340
object shape
44
4440
78
20
O+00
28
50:6
distractors object color
22:11
5640
67=11
Kitchen
0
Os0
Ozo
11:11
Pick
0=:
10
33
56::1
Kitchen
distractor
0.
20
67:
20
0+0
61
78:10
not only benefit from better Q-value estimation, but also leverage the standard deviation of the Q- values to measure the uncertainty of a state-action pair, i.e., whether is out-of-distribution or not. This serves as a test-time regularizer that equips the planner with the ability to balance exploitation and exploration without explicitly introducing conservatism training. By penalizing the actions leading to high uncertainty, we prioritize the actions that are more likely to achieve reliably high return. Formally, we modify the estimated return in Equation 2 of action sequence ag:ı be
R = =^ (Qo(zn. an) )un) + Σ (Ro(z,, a:)- u. std (10. (z., (z., a.)) (4)
where the uncertainty regularizer u, is highlighted red. Here, sid denotes the standard deviation operator, and is a constant coefficient that controls the regularization strength. We use the same value of À for both the offline and online stages in practice, but it need not be equal.
To facilitate rapid propagation of information acquired during finetuning, we maintain two replay buffers, Bor and B.n. for offline and online data, respectively, and optimize the objective in Equation on mini-batches of data sampled in equal parts from Boff and B.n. i.e., online interaction data heavily oversampled early in finetuning. Balanced sampling has been explored in various settings [33, 34. 35, 22. 36, 37], and we find that it consistently improves finetuning world models as well.
4 Experiments & Discussion
We evaluate method on diverse continuous control tasks from the D4RL [38] and xArm [39] task suites and quadruped locomotion in simulation, as well three visuo-motor control tasks real xArm robot, as visualized in Figure 3. Our experiments aim to answer the following questions:
Q1: How does our approach compare state-of-the-ar methods for offline RL and online RL?
Q2: Can approach be used finetune world models unseen tasks and task variations?
- Q3: How do individual components of our approach contribute to its success?
5

---

Table 3. Offline-to-oning results in simulation. Success rate (xArm) and normalized return (D4RL and quadruped) of methods before and after online finetuning. See Appendix B for task ex- planations. Mcan of 5 sceds.
^Am Transfer Tasks
Success Robo (84)
80
online
oflline-to-online
25
Task
TD-MPC
TD-MPC (e):
TD-MPC (40)
IQL
Ours
Push on
69.0
76.0
14.0 - 77.0
39.0
51.0
35.0
79.0
20
20
50
Push one
69.0
81.0
59.0
80.0
45.0
64.0
Enderonment (X100)
Pick ino
0.0
84.0
0.0 - 96.0
0.0 0.0-0. 0.0
0.0 88.0
Ours
TD-MPC
Pick
52.0
0.0058.0
0,0-100 1.0
0.0 66.0
Hopper on
4.8
2.8
0.8 11.0
66.3 - 76.1
49.6 100.7
Figure 4. Finetuning to unseen tasks. Success rate (%) aggre-
Hopper
6.1
13.4-, 13.7
76.5
1014
84,4 93.5
AutMazee imp)
0.0
52.0
0.0 68.0
54,0
80.0
58.0
96.0
AmiMaze imd))
72.0
91.0
62.0
88.0
75.0
89.0
gated across 9 transfer tasks in
Walk
8.8
9.2
18.5 - 1.2
19.1
19.2
67.2
85.8
simulated xArm environments.
Average
17.4
48.3
11.8 55.1
37.7 48.7
46.0 84.7
Mean of 5 seeds.
In the following, we first detail our experimental setup, and then proceed to address each of the above questions based our experimental results.
Real robot setup Our setup shown in Figure (right). The agent controls an xArm robot with jaw gripper using positional control, and 224 x 224 RGB image observations are captured by static third-person Intel RealSense camera (an additional top-view camera is used for kitchen; see Appendix B.1 for details); the agent also has access to robot proprioceptive information. Our setup requires no further instrumentation. We consider three tasks: reach, pick, and kitchen, and several task variations derived from them. Our tasks are visualized in Figure 3. The goal in reach is to reach a target with the end-effector, the goal in pick is to pick up and lift a target object above a height threshold, and the goal in kitchen is to grasp and put a pot in sink. We use manually designed detectors to determine task success and automatically provide sparse rewards (albeit noisy) for both offline and online RL. We use 120 offline trajectories for reach, 200 for pick, and 216 for kitchen; see Section 4.1 (Q3.2) 2) for details and ablations.
Simulation tasks and datasets We consider diverse set of tasks and datasets, includ- ing four tasks from the D4RL [38] benchmark (Hopper (medium). Hopper (medium-replay), AntMaze (medium-play) and AntMaze (mediun-diverss)), two visuo-motor control tasks from the xArm [40] benchmark (push and pick) and quadruped locomotion task ( (Walk); the two xArm tasks are similar to our real-world tasks except using lower image resolution (84 x 84) and dense rewards. See Figure 3 (left) for task visualizations. We also consider two dataset variations for each xArm task: medium, which contains 40k transitions (800 trajectories) sampled from suboptimal agent, and medium-replay which contains the first 40k transitions (800 trajectories) from the replay buffer of training TD-MPC agent from scratch. See Appendix B for more details.
Baselines We compare our approach against strong online RL and offline RL methods: (i) TD- MPC [26] trained from scratch with online interaction only, (ti) TD-MPC (+data) which utilizes the offline data by appending them to the replay buffer but is still trained online only, (iii TD-MPC (+offline) which naively pretrains on offline data and is then finetuned online, but without any of our additional contributions, and (iv) IQL. [19], ofate-of-the-ari offline RI. algorithm which has strong offline performance and also allows for policy improvement with online finetuning, See Appendix C and D for extensive implementation details on our method and baselines, respectively.
4.1 Results
Q1: Offfine-to-online RI. We benchmark methods across all tasks considered; real robot results are shown in Table and simulation results are shown in Table 3. We also provide aggregate curves in Figure (top) and per-task curves in Appendix A. Our approach consistently achieves strong zero- shot and online finetuning performance across tasks, outperforming offline-to-online TD-MPC and IQL by large margin in both simulation and real in terms of asymptotic performance. Notably, the performance of our method is more robust to variations in dataset and task than baselines, as evidenced by the aggregate results.

---

Q2: Finetuning to unseen tasks A key motivation for developing learning- based (and model- based in particular) methods for robotics is the potential for generalization and fast adaptation to unseen scenarios. To assess the versatility of our approach, we conduct additional offline-to-online finetuning experiments where the offline and online tasks are distinct, e.g,, transferring a reach policy to push task, introducing distractors, or changing the target ob- ject. We design 5 real-world transfer tasks, and 11 in simulation (9 for xArm and 2 for locomotion). See Figure 3 (center) and Appendix B for task visualizations and descriptions. Our real robot results are shown in Table 2, and aggregated simulation results are shown in Figure 4. We also provide per-task results in Appendix A. Our method successfully adapts to unseen tasks and task variations- both in simulation and in real and signif- icantly outperforms TD-MPC trained from scratch on the target task. However, as evidenced in the object color experiment of Table 2, transfer might not succeed if the initial model does not achieve any reward.
D4RL Tosks
Nomollaed Storo
75 25-
-500
-250
500
x/rm
100-
S (%)
60-
25
Envirommntt staps (K10P)
Q3: Ablations To understand how individual components
Ours
TD-MPC (+data)
contribute to the success of our method, we conduct series of ablation studies that exhaustively ablate each design choice. We highlight our key findings based on aggregate task results but note that per-task curves are available in Appendix A.
IQL
TD-MPC (+offline)
TD-MPC
D4RL Tasks
100
Normelieed Store Score
Q3.1: Algorithmic components We ablate each component of our proposed method in Figure 5 (botlom), averaged across all D4RL tasks. Specifically, we ablate (2) learning Qo with in-sample actions and expectile regression as described in Sec- tion 3.1, (2) using an ensemble of 5 value functions instead of as the original TD-MPC, (3) regularizing planning with our uncertainty penalty described in Section 3.2, and (4) using bal- anced sampling, i.e., sampling offline and online data in equal parts within each mini-batch. The results highlight the effective- ness of our key contributions. Balanced sampling improves the data-efficiency of online finetuning, and all other components contribute to both offline and online performance.
-250
Envronment steps (K10P)
Ours
wo in-sample learning
wlo value ensemble
wlo uncertainty penalty
w/o balanced sampling
Aggregate results. Top: comparison to baselines. Bottom: ablations. Offline pre- training is shaded gray. 5 seeds.
Figure 5.
Q3.2: Offline dataset Next, we investigate how the quantity and source of the offline dataset affect the success of online finetun- ing. We choose to conduct this ablation with real-world data to ensure that our conclusions generalize realistic robot learning scenarios. We experiment with two data sources and two dataset sizes: Base that consists of 50/100 tra- jectories (depending on the dataset size) gen-
Table 4. Real-world ablation on offline data. Success rate (%) as a function of online finetuning trials for two data sources and sizes from real- world reach task. Mean of 18 trials and seeds.
TD-MPC
Ours
Base
Diverse
Base
Diverse
Trials
100
120
50
70
120
0=o
50
Oza
0a0
33-00
72:16
crated by BC policy with added noise; and
10
44112
67112
50
6126
61
94.16
20
8316
78112
61
8910
89_0
89.0
Diverse that consists of the same 50/100 tra-
jectories as Base, but with an additional 20 exploratory trajectories from a suboptimal RL agent. In fact, the additional trajectories correspond to a 20-trial replay buffer from an experiment conducted in the early stages of the research project. Results for this experiment are shown in Table 4. We find that results improve with more data regardless of source, but that exploratory data holds far greater value: neither TD-MPC nor our method succeeds zero-shot when trained on the Base dataset - reeaardless of data quantity - whereas our method obtains 33% and 72% success rate with 70 and 120 trajectories, respectively, from the Diverse dataset. This result demonstrates that replay buffers from previously trained agents can be valuable data sources for future experiments.
7

---

Q3.3: Uncertainty regularization We seek to understand how the uncertainty regularization strength À influences the test- time performance of our method. Results for two tasks are shown in Figure 6; see Appendix A for more results. While > O almost always outperforms À 0 in both offline and online RL., large values, e.g-, À = 20, can be detrimental on- line RL in some cases. We use the same À for offline and online stages in this work, but remark that our uncertainty regularizer is a test-time regularizer, which can be tuned at any time at no cost.
Hopper (madiuum)
100
Nomrdbord Store
30-
-500
-200
250
600
Antilaae (mediuu--diverre
78-
Normeiaed
5 Related Work
20-
Offiine RL algorithms seek to learn RL policies solely from pre- existing datasets, which results in a state-action distribution shift between training and evaluation. This shift can be mitigated by applying explicit regularization techniques to conventional on- line RL algorithms, commonly Soft Actor-Critic (SAC; [41]). Most prior works constrain policy actions to be close to the data [13, 14, 15, 16], or regularize the value function to be conserva- tive (i.e,, underestimating) [17, 18, 42, 43]. While these strate- gies can be highly effective for offline RL, they often slow down convergence when finetuned with online interaction [21, 22, 24). Lastly, Agarwal et al. [44] and Yarats et al. [23] show that online RL algorithms are sufficient for offline RL when data is abundant.
250
Envrontent steps (x10')
A
A=
==
2
Figure 6. Regularization (A).
Score as function of regulariza- tion strength for two tasks from D4RL. Gray shade indicates of- fline RL. Mean of 5 seeds.
Finetuning policies with RL Multiple works have considered finetuning policies obtained by, e.g. imitation learning [45, 12, 36], self-supervised learning [46], or offline RL [21, 47, 22, 48, 24]. Notably, Rajeswaran et al. [45] learns model-free policy and constrains it to be close set of demonstrations, and Lee et al. [22] shows that finetuning an ensemble of model-free actor-critic agents trained with conservative Q-learning simulated control tasks. By instead learning world model on offline data, our method regularizes actions at test-fime (via planning) based on model uncertainty, without explicit loss terms, which is particularly beneficial for few-shot finetuning. While we do not compare to [22] in our experiments, we compare to IQL [19], a concurrent offline RI. method that conceptually closer to our method.
offline dataset can improve sample-efficiency in
Real-world RL Existing work on real robot learning typically trains policies on large amounts of data in simulation, and transfers learned policies to real robots without additional training (imulation-to-real) This introduces a domain gap, for which an array of mitigation strategies have been proposed, including domain randomization [8, 4, 1]. data augmentation [39], and sys- tem identification [49]. Additionally, building accurate simulation environments can be daunting task. At present, only limited number of studies have considered training RL policies in the real world without any reliance on simulators [50, 21, 51]. For example, researchers have proposed to accelerate online RL with human demonstrations [50] or offline datasets [21, 34] for model-free algorithms. Most recently, Wu et al. [51] demonstrates that MBRL can be data-efficient enough to learn diverse real robot tasks from scratch. Our work is conceptually similar to [21, 51] but is not directly comparable, as we consider an order of magnitude less online data than prior work.
Limitations
Several open problems remain: offline data quality and quantity heavily impact few-shot learning, and we limit ourselves to sparse rewards since dense rewards are difficult to obtain in the real world. We also find that the optimal value of À can differ between tasks and between offline and online RL. We leave it as future work to automate this hyperparameter search, but note that doing so is relatively cheap since it can be adjusted at test-time without any overhead. Lastly, we consider pretraining on a single task and transferring to unseen variations. Given such limited data for pretraining, some structural similarity between tasks is necessary for few-shot learning to be successful.
8

---

Acknowledgments
This work was supported, in part, by the Amazon Research Award, gifts from Qualcomm and the Technology Innovation Program (20018112, Development of autonomous manipulation and grip- ping technology using imitation learning based on visual and tactile sensing) funded by the Ministry of Trade, Industry & Energy (MOTIE), Korea.
References
[1] M. Andrychowicz, B. Baker, M. Chociej, Józefowicz, B. McGrew, J. W. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider. Sidor, J. Tobin, Welinder,
L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. The International Jour-
nal of Robotics Research (IJRR), 39:20 - 3, 2018.
[2] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer,
S. Hashme, C. Hesse, R. Józefowicz, S. Gray, C. Olsson, J. W. Pachocki, M. Petrov, H. P. de Oliveira Pinto, J. Raiman, Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, Tang, Wolski, and S. Zhang. Dota 2 with large scale deep reinforcement learning. ArXiv, abs/1912.06680, 2019.
[3] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, T. P. Lillicrap. and D. Silver. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588:604 609, 2019.
[4] X. B. Peng, M. Andrychowicz, W. 7aremba, and Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1-8, 2017.
[5] L. Pinto, M. Andrychowicz, Welinder, W. Zaremba, and Abbeel. Asymmetric actor critic for image-basee basge- robot learning. Proceedings of Robotics: Science and Systems (RSS), 2018.
[6] F. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-7: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning (CoRL), 2022.
[7] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. C. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. S. Ryoo, G. Salazar, P. R. Sanketi, K. Sayed, J. Singh, S. A. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. H. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. RT-1: Robotics transformer for real-world control at scale. ArXiv, abs/2212.065
2022.
[8] Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and Abbeel. Domain randomiza- tion for transferring deep neural networks from simulation the real world. 2017 JEEE/RS] International Conference Intelligent Robots and Systems (IROS), pages 23 2017.
[9] W. Zhao, J. P. Queralta, and T. Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: survey. 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pages 737-744, 2020.
[10] N. Hansen, R. Jangir, Y. Sun, G. Alenyå, P. Abbeel, A. A. Efros, L. Pinto, and X. Wang. Self- supervised policy adaptation during deployment. International Conference on Learning Representations (ICLR), 2021.
[11] A. Kumar, J. Hong, A. Singh, and S. Levine. Should I run offline reinforcement learning behavioral cloning? In International Conference Learning Representations (ICLR), 2022.

---

[12] B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (VPT): Learning to act by watching unlaheled online videos. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[13] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning (ICML), 2018.
[14] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. ArXiv, abs/1911. 1361, 2019.
[15] R. Kidambi, A. Rajeswaran, Netrapalli, and T. Joachims. MOReL: Model-based offline reinforcement learning. Advances Neural Information Processing Systems (NeurIPS), 33: 21810-21823, 2020.
[16] S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems (NewrIPS), 14-20132-201444 2021.
[17] A. Kumar, Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[18] A. Kumar, Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning. Advances in Neural Information Processing Systems (NeurIPS), 33:1 179-1191, 2020.
[19] Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations (ICLR), 2022.
[20] S. M. Richards, N. Azizan, J.-J. Slotine, and M. Pavone. Adaptive-comtrolorrinttee meta- learning for nonlinear systems. Proceedings of Robotics: Science and Systems (RSS), 2021.
[21] A. Nair, A. Gupta, M. Dalal, and S. Levine. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv preprint M0Xiv:000609335 2020.
[22] S. Lee, Y. Seo, K. Lee, Abbeel, and Shin. Offline-to-onlin reinforcement learning via balanced replay and pessimistic Q-ensemble. In Conference Robot Learning (CoRI.), pages 1712. PMLR, 2022.
[23] D. Yarats, D. Brandfonbrener, H. Liu, M. Laskin, P. Abbeel, A. Lazaric, and L. Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. ArXiv, abs/2201. 13425,, 2022.
[24] M. Nakamoto, Y. Zhai, A. Singh, M. S. Mark, Y. Ma, C. Finn, A. Kumar, and S. Levine. Cal- QL: Calibrated offline RL pre-training for efficient online fine-tuning. ArXiv, abs/2303.05479, 2023.
[25] D. Ha and J. Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems 31, pages 2451-2463. Curran Associates, Inc., 2018.
[26] N. Hansen, X. Wang, and H. Su. Temporal difference learning for model predictive control. In International Conference Machine Learning (ICML), 2022.
[27] L. P. Kaelbling, M. L. Littman, and A. Cassandra. Planning and acting in partially observ- able stochastic domains. Artificial Intelligence, 1998.
[28] G. Williams, A. Aldrich, and E. A. Theodorou. Model predictive path integral control using covariance variable importance sampling. ArXiv, abs/1550.00 149, 2015.
[29] J. Peters and S. Schaal. Reinforcement learning by reward-weighte regression for operational space control. In International Conference on Machine Learning (ICML), 2007.
10

---

[30] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 19.000/777 2019.
[31] B. Charpentier, R. Senanayake, M. Kochenderfer, and S. Günnemann. Disentangling epistemic and aleatoric uncertainty in reinforcement learning. arXiv preprint arXiv:2206.0/5555 2022.
[32] X. Chen, C. Wang, Z. Zhou, and K. W. Ross. Randomized ensembled double q-learning: Learning fast without model. In International Conference Learning Representations (ICLR), 2021.
[33] R. C. Julian, B. Swanson, G. S. Sukhatme, S. Levine, C. Finn, and K. Hausman. Efficient adaptation for end-to-end vision-based robotic manipulation. ArXiv, abs/2004. 10190, 2020.
[34] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. MT-Opt: Continuous multi-task robotic reinforcement learning scale. ArXiv, abs/2104.08212, 2021.
[35] N. Hansen, H. Su, and X. Wang. Stabilizing deep Q-learning with convnets and vision transformers under data augmentation. Advances in Neural Information Processing Systems (NeurIPS), 34:3680-366 -090000-63--20000 2021.
[36] N. Hansen, Y. Lin, H. Su, X. Wang, V. Kumar, and A. Rajeswaran. Modem: Accelerating visual model-based reinforcement learning with demonstrations. In International Conference on Learning Representations (ICLR), 2023.
[37] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T.. Springenberg, T. Becles, J. Bruce, A. Razavi, A. D. Edwards, N. M. O. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. A generalist agent.
Transactions on Machine Learning Research, 2022.
[38] Fu, A. Kumar, O. Nachum, G. Tucker, and Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint ar arXiv:2004.0022 2020.
[39] R. Jangir, N. Hansen, S. Ghosal, M. Jain, and X. Wang. Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters, pages 1-1, 2022.
[40] Y. Ze, N. Hansen, Chen, M. Jain, and X. Wang. Visual reinforcement learning with self- supervised 3D representations. IEEE Robotics and Automation Letters, pages 1-1, 2023.
[41] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine. Soft actor-critic algorithms and applications. ArXiv, abs/18 181.00905, 2018.
[42] T. Yu, G. Thomas, L. Yu, S. Ermon, Zou, S. Levine, C. Finn, and T. Ma. MOPO: Model-based offline policy optimization. Advances Neural Information Processing Systems (NeurIPS), 2020.
[43] A. Kumar, A. Singh, F. Ebert, Y. Yang, C. Finn, and S. Levine. Pre-training for robots: Offline RL enables learning new tasks from handful of trials. arXiv preprint 1rXiv:2210.05178. 2022.
[44] R. Agarwal, D. Schuurmans, and M. Norouzi. Striving for simplicity off-policy deep rein- 2019.
forcement learning, ArXiv,
[45] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, Schulman, E. Todorov, and S. Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demon- strations. In Proceedings of Robotics: Science and Systems (RSS), 2018.

---

[46] L. Ouyang, J. Wu, X. Jiang, D. Almeida. C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Gray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welin- der, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In A. H. Oh, A. Agarwal, Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems (NeurIPS). 2022.
[47] C. Wang, X. Luo, K. W. Ross, and D. Li. VRL3: A data-driven framework for visual deep reinforcement learning. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems (NeurIPS), 2022.
[48] Y. Xu, N. Hansen, Z. Wang, Y.-C. Chan, H. Su, and Z. Tu. On the feasibility of cross-task transfer with model-based reinforcement learning. In International Conference on Learning Representations (ICLR), 2023.
[49] A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid motor adaptation for legged robots. 2021.
ArXiv,
[50] A. Zhan, Zhao, L. Pinto, P. Abbeel, and M. Laskin. A framework for efficient robotic manipulation. ArXiv, abs/2012.07975, 2020.
[51] Wu, A. Escontrela, D. Hafner, K. Goldberg, and Abbeel. Daydreamer: World models for physical robot learning. Conference on Robot Learning (CoRL), 2022.
[52] I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations (ICLR), 2021.
[53] T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay. arXiv preprint 1rX:v1551.05995, 2015.
12

---

A Additional Results
In addition to the aggregated results in the main paper, we also provide per-task results for the experiments and tasks simulation. Our benchmark results are shown in Figure 7, and task transfer results are shown Figure 10. Per-task results for ablations are shown Figure and Figure 9.
Hoppor
AntMaaa (medium--pig) play))
Push (medium)
100
Normal Boore
75
Narnal Soore
Success Rate MNe
75
25-
25-
-500
250
-500
-250
500
-50
-25
Emvrrnmmnn otrppe 1111)
Finsrrnnmen steps Ex 10%)
Emvironment adeep (XC10))
Hopper (medium-eppee))
AntMaze (medRarm-dvveree)
(mediun-eppey)
Nometted Soore
75
75
Success Rate (5)
75
60-
Nomal
60
60-
-500
-250
250
500
-000
-250
200
500
-00
-25
25
50
Emvronment (x10')
Envirommen
Emvirommnnt (XC10))
Pick (medium)
Pick (meciiu-essiay)
Walk
100
100
100
Sterras Raain 1841
Nomolized Soom
75
Succosss Rata
80
50
.00
-50
-25
-25
-25
(x10%)
(x102)
Ours
TD-MPC
TD-MPC (+data)
TD-MPC (+affline)
IQL
Figure 7. Comparison of our method against baselines. Offline pretraining is shaded gray. Mean of 5 seeds; shaded area indicates 95% Cls.
13

---

Push (mediums replay)
Hoppar (medium)
AntMaze (medium playy)
100
Surzaen Rode (S)
75
Normalized Stoee
75
Normaltzed Scoees
75-
80
Plck (medium roploay
Hozper imodium repley!
AntMaaz (medium divvereee
100
100
100-
Dban (8)
75-
Normelzed Sioee
Normaizee Sioee
75-
25
25
25-
M J
-250
Emvrrnmmnn stops (x10))
Environment steps (x
Envrronmen steps (X10")
Ours
w/o in-sample leaming
w/o uncertainty penalty
w/o value ensemble
wlo balanced sampling
Figure 8. Per-task ablation results. Offline pretraining is shaded gray. Mean of 5 seeds; shaded area indicates 95% Cls.
Push (meliam-rrepaay)
Hopper (redium)
AntMaaee (mediurn-pley)
100
100
100
Success (8)
70-
Nommkkzee Score
Nommstzed Score
77-
60-
25-
25-
Plik (medium-repley)
Hopper (medium--eepeee)
AntMo..e Cowediam-dinnn
100
100
100-
Succees (5)
Nemmkkzee Soore
Nommszzet Sezrr
60-
60
500
25-
25-
2500
são
Environment stape (x10%)
Enviranment staps (x10
Environment staps (X30")
A=1
A=3
A=
=
Figure 9. Ablation study on uncertainty coefficient (A). Offline pretraining is shaded gray. Mean of seeds; shaded area indicates 95% Cls.
14

---

Cyändar
Cyliner
76-
Surpan Rate
60-
Sphorn Lighting
Sphare Lightng
Cylnndrr
Suctwas Iose (S]
78
50-
Cube Push Cylinder Target
Sustoess Iose (5]
-0
-30
-00
-10
10
20
30
40
50
40
30
-2
0
10
20
30
40
00
-00
-30
20
-10
10
20
40
00
Envemement
(x10')
Enveromment
(x10'))
Envirommet
(x10'))
Ours
TD-MPC
Walk Walk Fast
Walk Walk Rugged Terrain
100
100
Nomaltoo Boore
75
Nor Bcom
75
50
50
25-
75
25
1.5
Envtrommnt stops
stops (410")
Ours
TD-MPC
Figure 10. Task transfer results. Success rate (%) of our method and TD-MPC trained from scratch on all simulated transfer tasks. The first nine are designed based on the xArm [40] task suite, and the last two are quadruped locomotion. Offline pretraining is shaded gray. Mean of 5 seeds; shaded area indicates 95% Cls.
15

---

B Tasks and Datasets
B.1 Real-World Tasks and Datasets
We implement three visuo-motor control tasks, reach, pick and kitchen on UFactory xArm 7 robot arm. Here we first introduce the setup for reach and pick, which share the same workspace. We use an Intel RealSense Depth Camera D435 as the only external sensor. The observation space contains a 224 224 RGB image and an 8-dimensional robot proprio- ceptive state including the position, rotation, and the opening of the end-effector and a boolean value indicating whether the gripper is stuck. Both tasks are illustrated in Figure (second from the left). For safety reasons, we limit the moving range of the gripper in 30cm x 30cm x 30cm cube, of which pro- jection on the table is illustrated in Figure 11. To promote consistency between experiments, we evaluate agents on a set of fixed positions, visualized as red crosses in the aforemen- tioned figure. The setup for kitchen is shown in Figure 12. We use two D435 cameras for this task, providing both front view and top view. The observation space thus contains two 224 x 224 RGB images and the 8-dimensional robot proprio- ceptive state.
Figure 11. Real-world workspace. Moving range of the end-effector and the initialization range of tar- get/object are shaded in the image. The positions for evaluation are la- beled by crosses.
(a) Kitchen setup
(b) Front view
(c) Top view
Figure 12. Real-world kitchen task setup. (a) Setup of the kitchen workspace with the xArm robot. (b)-(c) Sample images from the front view and the top view, respectively.
Below we describe each task and the data used for offline pretraining detail. Figure 13 shows sample trajectories for these tasks.
Reach The objective of this task is to accurately position the red hexagonal prism, held by the gripper, above the blue square target. The action space of this task is defined by the first two dimen- sions, which correspond to the horizontal plane. The agent will receive a reward of when the object successfully placed above the target, and reward of otherwise. The offline dataset for reach comprises 100 trajectories collected using behavior-cloning policy, which exhibits an approximate success rate of 50%. Additionally, there are 20 trajectories collected through teleoperation, where the agent moves randomly, including attempts to cross the boundaries of the allowable end-effector movement. These 20 trajectories are considered to be diverse and are utilized for conducting an ablation study around the quality of the offline dataset.
The objective of this task is grasp and lift a red hexagonal prism by the gripper. The action space of this task contains the position of the end-effector and the opening of the gripper. The agent will receive a reward of 1 when the object is successfully lifted above height threshold, 0.5 when the object is grasped but not lifted, and 0 otherwise. The offline dataset for pick comprises 200 trajectories collected using a BC policy that has an approximate success rate of 50%.
Pick
16

---

Kitchen This task requires the xArm robot to grasp a pot and put it into sink in a toy kitchen environment. The agent will receive a reward of when the pot is successfully placed in the sink, 0.5 when the pot is grasped, and otherwise. The offline dataset for kitchen consists of 216 trajectories, of which 100 are human teleoperation trajectories, 25 are from BC policies, and are from offline RL policies.
Real-world transfer tasks We designed two transfer tasks for both reach and pick, and one for kitchen as shown in Figure 3 (the second from right). As the red hexagonal prism is important indicator of the end-effector position in reach, we modify the task by (1) placing an additional red hexagonal prism on the table, alongside the existing one, and (2) replacing the object with small red ketchup bottle, whose bottom not aligned with the end-effector. pick, the red hexagonal prism regarded as target object. Therefore we (1) add two distractors, each with distinct shape and color compared to the target object, and (2) change the color and shape of the object (from red hexagonal prism to green octagonal prism). For kitchen, we also add teapot with similar color distractor. We've shown by experiments that different modifications will have different effects on subsequent performance in finetuning, which demonstrates both the effectiveness and limitation of the offline-to-online pipeline we discussed.
as the pot in the scene
initial state
highlight
highlight
termination
initial state
highlight 1
highlight
termination
Reach
Pick
Kitchen
Figure 13. Sample trajectories. We include eleven trajectories from the offline dataset or evaluation results, which illustrate all wellwwrldd tasks considered in this work. Successful trajectories are marked green while failed trajectories are marked red.
17

---

B.2 Simulation Tasks and Datasets
xArm Push and pick are two visuo-motor control tasks in the xArm robot simulation environ- ment [40] implemented in MuJoCo. The observations consist of an 84 x 84 RGB image and a 4-dimensional robot proprioceptive state including the position of the end-effector and the opening of the gripper. The action space the control signal for this 4-dimensional robot state. The tasks visualized in Figure 3 (left). push requires the robot push green cube to the red target. The goal in pick is to pick up cube and lift it above height threshold. Handcrafted dense rewards are used for these two tasks. We collected the offline data for offline-to-online finetuning experiments by training TD-MPC agents from scratch on these tasks. The medium datasets contain 40k transi- tions (800 trajectories) sampled from a sub-optimal agent, and the medium-replay datasets contain the first 40k transitions (800 trajectories) from the replay buffers. Figure 14 gives an overview of the offline data distribution for the two tasks.
(madium-replay)
(medum-reploa))
70-
60-
50
40-
40
Count
50-
30-
20-
10-
Push (medium)
Pick (mediu))
70
60
50-
50-
40
Coamt
30-
10
10
15
-30
-25
15
0
5
10
15
25
30
Eplcode Return
Estsode Return
Figure 14. Offline dataset statistics for xArm tasks in simulation. We plot the distribution of episode returns for trajectories in the two offline datasets. The red line indicates the mean perfor- mance achieved by our method after online finetuning.
Quadruped locomotion Walk is state-only continuous control task with a 12-DoF Unitree Gol robot, as visualized in Figure 3 (left). The policy takes robot states as input and output control signal for 12 joints. The goal of this task to control the robot to walk forward at a specific velocity. Rewards consist of a major velocity reward that is maximized when the forward velocity matches the desired velocity, and minor component that penalizes unsmooth actions.
We designed nine transfer tasks based on reach (the same task as real reach but simplified because of the knowledge of ground-truth positions) and push with xArm, and two trans- fer tasks with the legged robot in simulation to evaluate the generalization capability of offline pretrained models. Compared to real-world tasks, the online budget is abundant in simulation, thus we increase the disparity between offline and online tasks such as finetuning on totally different task. As the target point for both xArm tasks is red circle, we directly use reach as offline pretrain task and online finetuning cylinder, and push cube with an obstacle. For quadruped locomotion, we require the robot to walk at higher target speed (twice the pretrained speed) and to walk on new rugged terrain. Tasks are illustrated in Figure 15.
Transfer tasks
different instances of push including push cube, push sphere, push
We consider four representative tasks from two domains (Hopper and AntMaze) in the D4RL [38] benchmark. Each domain contains two data compositions. Hopper is a Gym locomotion domain where the goal is to make hops that move in the forward (right) direction. Observations
D4RL
18

---

contain the positions and velocities of different body parts of the hopper. The action space is a 3- dimension space controlling the torques applied on the three joints of the hopper. Hopper (medium) uses 1M samples from a policy trained to approximately 1/3 the performance of the expert, while Hopper (medium-replay) uses the replay buffer a policy trained up to the performance of medium agent. Antmaze navigation domain with a complex 8-DoF quadruped robot. We use the medium maze layout, which is shown in Figure (left). The play dataset contains 1M samples generated by commanding specific hand-picked goal locations from hand-picked initial positions, and the diverse dataset contains IM samples generated by commanding random goal locations in the maze and navigating the ant to them. This domain is notoriously challenging because of the need to "stitch" suboptimal trajectories. These four tasks are officially named mamaazz-mddiun-llayy. and antmaze sorrzze-mediin -ndiivers--2222
pepper--edii--rpllla----
the D4RL benchmark.
C Implementation Details
Q-ensemble and uncertainty estimation We provide PyTorch-style pseudo-code for the imple- mentation the Q-ensemble and uncertainty estimation discussed in Section 3.2. Here Qs is a list of Q-networks. We use the minimum value of two randomly selected Q-networks for Q-value es- timation, and the uncertainty estimated by the standard deviation of all Q-values. We five Q- Q-networks in our implementation.
def Q_estimate (Qs, a):
torch. cat ( [z, a] dim=-1) concatemate (latent) state and action idxs raadom_choice (Qe) 2, replace=Palse) randomly select two distinct Qs q1, q2 Qs [idxs [o]] (x), Qs [idxs
return torch. torch.min(q/) (in q2) # return the minimum of the two as Q value estimation
def Q_ Q_uncertainty(Qe, z, a):
x torch, cat cat a], dim=-1) # concatemate (latent) state and action qe = torch. stack for in Qa), dim=0) uncertainty
(dim=0) # compute the standard deviation as uncertaenty return uncertainty
Network architecture For the real robot tasks and simulated xArm tasks where observations tain both an RGB image and robot proprioceptive state, we separately embed them into feature vectors of the same dimensions with a convolutional neural network and a 2-layer MLP respec- tively, and do element-wise addition to get a fused feature vector. For real-world kitchen tasks, where observations include two RGB images and proprioceptive state, we use separate encoders embed them into three feature vectors and do the element- wise addition. For D4RL and quadruped locomotion tasks where observations Q-networks to implement the Q-ensemble for uncertainty estimation. All Q-networks have the same architecture. An additional V network is used for state value estimation discussed in Section 3.1.
state features, only the state encoder is used. We five
Hyperparameters We list the hyperparameters of our algorithm in Table 5. The hyperparameters related to our key contributions are highlighted
Other details We apply image shift augmentation [52] image observations, and use Prioritized Experience Replay (PER; [53]) when sampling from replay buffers.
D Baselines
TD-MPC We use the same architecture and hyperparameters for our method and our three TD-MPC baselines in the public TD-MPC implementation from https://github com/ ni nicklashansen// except that multiple encoders used to accommodate both visual inputs and robot proprioceptive information in the real robot and xArm tasks, described in Apppendix C.
19

---

For the TD-MPC (+data) baseline, we append the offline data to the replay buffer at the beginning of online training so that they can be sampled together with the newly-collected data for model up- date. For the TD-MPC (+offline) baseline, we naively pretrain the model on offline data and then finetune with online RL without any changes to hyperparameters.
IQL We use the official implementation from https:/ //ts::///ii /egggttthuu...o http::////ggittb....ookoottrrtt/ http.///o.kotttkkoo///iiiiiiii q q_learning for the IQL baseline. We the same hyperparameters that the authors used for D4RI. tasks. For xArm tasks, we perform grid search over the hyperparameters 7 (0.5, 0.6, 0.7, 0.8, 0.9, 0.95) and B E (0.5, 1.0, 3.0, 10.0), and we find that expectile 5 = 0.95 and temperature 3 10.0 achieves the best results. We add the same image encoder as ours to the IQL implementation in visuo- motor control tasks.
Table 5. Hyperparameters.
Hyperparameter
Value
Expectile (T)
0.9 (AntMaze, xArm, Walk)
0.7 (Hopper)
AWR temperature (3)
10.0 (AntMaze)
3.0 (Hopper, xArm)
1.0 (Walk)
Uncertainty coefficient (A)
1 (xArm, Walk)
3 (AntMaze)
20 (Hopper)
Q ensemble size
5
Batch size Learning rate Optimizer Discount
256
3e-4
Ad 0.9, 32 0.999)
0.99 (D4RL, Walk)
0.9 (xArm)
Action repeat
1 (D4RL, Walk)
2 (xArm)
Value loss coefficient
0.1
Reward loss coefficient
0.5
Latent dynamics loss coefficient
20
Temporal coefficient
0.5
Target network update frequency
2
Polyak
0.99
MLP hidden size
512
Latent state dimension
50
Population size
512
Elite fraction
50
Policy fraction
0.1
Planning iterations
6 (xArm, Walk)
1 (D4RL)
Planning horizon
5
Planning temperature
0.5
Planning momentum coefficient
0.1
20

---

(a) Reach.
(b) Push.
(c) Push sphere with increased lighting.
(d) Push cylinder.
(e) Push with obstacle.
0 Walk on rugged terrain.
Figure 15. Transfer tasks in simulation. We consider total of eleven transfer settings in sim- ulation. We here visualize trajectory for each of the tasks used our xArm experiments, and trajectory for walking on rugged terrain with the legged robot. Task labels correspond to those shown Figure 10.
21


Reviews:
[{"review_id": "rZRtKrim6m", "reviewer": "Reviewer_mpkG", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents an approach to offline-online approach for MBRL. The authors enable generalization in the presence of distribution shift from offline to online data by regularizing the policy using uncertainty on the value function as obtained from an ensemble. The proposed method is evaluated against IQL and TD-MPC baselines on both simulated and real-world robot data.", "summary_of_recommendation": "Overall, the paper is well written and an enjoyable, educational read. The use of epistemic uncertainty to regularize online RL is a good idea. The real-world experiments that show robust generalization to new settings are compelling. My biggest concerns with the paper center around the per-task performance in some of the experiments - where the baselines sometimes appear to do better. Also, the literature review should really cover epistemic uncertainty in RL. As it stands, I am still leaning towards accepting the paper assuming the literature review is made more complete.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "In addition to the weaknesses listed above, I am hoping the authors can address the following clarifications during the rebuttal period.\n1. The authors state in the second paragraph of the paper that extrapolation errors are less of an issue for online RL because they can learn from negative rewards. However, does this not come with safety concerns?\n2. Does the regularizer get used online and offline? I did not understand this point.\n3. Could you clarify why TD-MPC underperforms the proposed method in Q2? If I am understanding correctly, the proposed method is offline trained on task A and online finetuned on task B, while TD-MPC is just trained on task B. I am surprised that TD-MPC does worse on task B.\n4. Does the original TD-MPC paper also have an ensemble, albeit smaller than the one considered here? What are the key differences in how the ensembles are used?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n* The paper is very well written and structured. It was an enjoyable read!\n* The topic of using epistemic uncertainty to modulate online RL is timely and interesting. The problem is well-motivated and the literature review does a good job of contextualizing the paper in prior work.\n* The authors do their due dilligence in ablating the approach and conduct many real-world robot experiments.\n* I appreciate the candor and honesty with which the results are presented (both good and bad).\n* Further details and experiments in the appendix provide additional context for the method and the claims made in the main paper.\n\nWeaknesses:\n\n* The statement \"We are, to the best of our knowledge, the first work to investigate offline-to-online finetuning with MBRL on real robots\" is bold. From a quick search, I could not find a paper that does this exactly, but I would make sure to do a thorough literature review to ensure the accuracy of the statement. The literature review is also missing, for example, the following works:\n\n[A] A. Kumar et al. \"Pre-training for robots: Offline RL enables learning new tasks from a handful of trials.\" arXiv, 2022.\n[B] S. M. Richards et al. \"Adaptive-Control-Oriented Meta-Learning for Nonlinear Systems.\" RSS, 2021.\n\nSeparately, I would also recommend including in the literature review the use of epistemic uncertainty estimation for RL methods. For example, the following paper:\n\n[C] B. Charpentier et al. \"Disentangling epistemic and aleatoric uncertainty in reinforcement learning.\" arXiv, 2022.\n\n* My biggest concern about the paper is the presented results. The proposed method does not always outperform the baselines (e.g., in Table 3 - simulation, the proposed method performs the best in 2 of 6 tasks, but outperforms overall across tasks, in Fig. 5, from the first two plots, the method is not clearly better). The robot tasks are also pretty simple (e.g., reach and pick). That being said, I do agree that the method appears to be more robust than the baselines to task variations and new settings.\n\n* The references have some consistency issues. For example, conference acronyms are sometimes included (e.g., ICLR in [10]) and sometimes not (e.g., [11]). When possible, the conference venue should be cited instead of ArXiv. The references should in general be proofread (e.g., should capitalize \"RT-1\" in [7], \"I\" in [11], \"Q\" in [18], \"RL\" in [23], etc.).", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "eJ4x921hl5", "reviewer": "Reviewer_S26C", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this research, the authors propose a method to address the problem of inefficient data efficiency and challenging training of conventional reinforcement learning in real robots. They achieve this by pre-training a world model using offline data and then fine-tuning the model with online data, enabling effective policy learning for new tasks. Experiments on simulators and real robots demonstrate that the approach is capable of effective learning for both seen and unseen tasks even with limited offline data.", "summary_of_recommendation": "This paper demonstrates high performance with a small number of trials on a real robot, and I believe it is highly effective and important for future world model research. Therefore, I recommend accepting this paper for publication.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- In the paper, the authors state that \"However, as evidenced in the object color experiment of Table 2, transfer might not succeed if the initial model does not achieve any reward.\" Could the authors try increasing the number of trials to see if transfer becomes possible? If it becomes possible, how many trials would be needed to achieve it? I am concerned whether this issue represents an inherent limitation of the proposed method, and if it is a limitation, I would like to know why.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- The paper is well-structured and easy to read.\n- The proposed test-time behavior regularization in this study is simple yet experimentally shown to be highly effective. Its applicability to unknown tasks with only a few interactions in the real world, without relying on simulators, makes it particularly valuable.\n- The study demonstrates high credibility by conducting experiments not only with simulations but also with real robots, comparing state-of-the-art methods for offline RL and online RL. Additionally, the credibility is further reinforced by testing the proposed approach with variations in unseen tasks and changes in hyperparameters.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "7XSy5DLMWM", "reviewer": "Reviewer_LZqg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper suggests a set of adjustments to the TD-MPC model-based RL algorithm, and applies them to an offline-to-online setting applicable to robotic tasks. In particular, the paper suggests a regularization technique based on an estimate of the epistemic uncertainty of the dynamics model, obtained by training an ensemble of Q-value networks. This uncertainty weighting helps limit the effect of out-of-distribution transitions when fine-tuning a model trained on an offline dataset. ", "summary_of_recommendation": "While the additions made to the TD-MPC method seem sound and adequate for the scenarios discussed in the paper, they are not in themselves novel our groundbreaking.The results demonstrated on the various experiments are not as significantly dominant over baselines as claimed by the authors. While the paper describes an interesting method and the initial results do seem promising for fine-tuning models trained on offline data, further experimentation may be required in order for the results to be convincing. \n\nEDIT: after reading through the other reviews and considering the information and experiments added in the rebuttal, including the updated limitations section, I am inclined towards accepting this paper and have updated my score to \"weak accept\" accordingly. While I am still not fully convinced of the SoTA claims for the results, they do back the authors claims on the usability of their method; I believe this paper is a valuable addition to the robot learning community.  ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Looking at the per-task ablation study, seems like removing in-sample learning improves performance on the xArm tasks. Any intuition on why this happens?\n\nIn general, the ‘pick’ task seems to stand out, with the offline training not contributing to the fine-tuning process, and the uncertainty regularization hurting, rather than improving, performance. Is the offline data distribution here extremely different from the online distribution? \n\nWere ablations performed on the real-world tasks and experiments? Would be interesting to see how TD-MPC performs on the offline to online real world tasks (Table 1) with only some of the improvements proposed in this paper. \n\nLine 86: Transition function T is for full (unknown) states s, as mentioned on line 71. How come the hidden states z also obey it?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In general, I like the stated goals and aim of this paper - it makes sense to use offline datasets and model-based planning to solve robotic tasks in a more data-efficient manner (especially compared to model-free RL). Additionally, while not in themselves novel, the adjustments the authors make to the TD-MPC approach seem sound, and make sense in the offline-to-online fine-tuning setting the paper focuses on. The authors provide a set of experiments which covers multiple domains with a variety of dynamics and reward structures, and show that their method is at least competitive with existing approaches. \n\nThe paper is nicely organized and well written, and presents its claims clearly. Figures and tables in the paper are also clears and easy to understand. \n\n\nWhile the results presented in the paper seem promising, some concerns arise regarding the claims of “outperforming SoTA methods”. \nOne such concern is that the aggregated result plots provided in the main text may be somewhat misleading. As evident in the appendix (Figure 7), In both xArm tasks, it seems like the proposed method does not outperform the TD-MPC baseline; in most D4RL tasks, it seems like the proposed method is comparable to IQL, but does not significantly outperform it. As IQL was tuned specifically for the D4RL tasks, this raises concern that with proper hyperparameter tuning, IQL may outperform the method proposed in this paper on the xArm tasks as well. \nThis issue is also apparent in Table 3: the proposed method only outperforms both baselines on two specific simulated tasks in the offline-to-online setting. One wonders whether the baselines would perform better if their hyperparameters were tuned to the task they were not originally trained on (i.e. TD-MPC to the D4RL tasks, and IQL to the xArm tasks).\n\nRegarding the claims of adjustment to “unseen tasks”:\nIf a Q-value estimate is learned from the offline data, is this method only relevant when applied to tasks with similar reward structures? The method claims fine tuning to unseen tasks - however, this may be misleading if it only refers to unseen distractors in similar tasks. One of the main advantages of planning using world models is that they can be used for multiple tasks, regardless of the task objective (given by the reward); in this work, with the reward model learned along with the dynamics, this advantage is forfeit. \nThis design of the world model to include the value function may explain its success on the “unseen” tasks in the experiments - these are all pretty close to the training tasks, with rewards similar and only visual appearances of objects or background changing. While robustness to distractors is an important trait in world models for robots, it seems like this property is also only available when visual features do not change too much - the failure case of picking up the green object instead of the red one seems like one that a model should be able to solve with appropriate fine-tuning.\n\nIn order to cement a more robust demonstration of the claims made in the introduction, the experiments should include some scenarios where the reward changes significantly between the \"seen\" and \"unseen\" tasks. At the very least, this should be addressed in the limitations section. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "3eKA-b5NLO", "reviewer": "Reviewer_UN8h", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents offline training (fixed data) then online fine-tuning in several simulation tasks (D4RL, xArm), as well as real world experiments with an xArm performing pick and place. This paper introduces several improvements to TD-MPC [1]. The first is to include in-sampling of actions (from IQL [2]) to reduce overestimation errors in value function prediction. Secondly, an ensemble of Q functions are employed, both to improve Q function predictions, and to utilise the standard deviation of the ensembles as an uncertainty penalty added to the reward and the terminal Q value. One claim is that the uncertainty penalty can be used as a regularizer at test-time, retaining model expressiveness through training.\n\n[1] N. Hansen, X. Wang, and H. Su. Temporal difference learning for model predictive control. In\nICML, 2022.\n\n[2] Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning.\nICLR 2022\n", "summary_of_recommendation": "While in general I like the ideas introduced (bringing together of several previous works), the results don't seem overly significant. The real world experiments are promising.\n\nAfter considering the additional information provided in the rebuttal I have increased my recommendation to a weak accept. Thank you to the authors for their efforts.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "In the transfer tasks, Figure 4 and Figure 10, TD-MPC is trained from scratch whereas the method from this paper is trained on offline data first (from a different task?). These don’t seem like a fair comparison. Is there a reason we don’t see TD-MPC (+offline) for the same offline datasets?\n\nWhat is a trial? Is this a fixed number of environment steps or termination?\n\nHow is the perception encoder trained (h ϕ)? \n\nDouble check references include the conference citation if accepted, not just the preprint:\nE.g: \nVideo PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos -> NeurIPS 22\nMOReL: Model-Based Offline Reinforcement Learning -> NeurIPS 2020\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper was very well presented and well written. In general the idea of training on a dataset of previous experience then fine tuning on the robot is very appealing, especially for a small amount of new data and considering model uncertainty at test time. The results show improvements over existing methods on the real robot. The ablations effectively highlight the importance of each improvement. It was good to see that previously collected real-world data improved zero shot transfer (Q3.2), but also good to see similar performance across datasets with more interaction. \n\n“Our approach consistently achieves strong zero-shot and online finetuning performance across all tasks, outperforming offline-to-online TD-MPC and IQL by a large margin in both simulation and real in terms of asymptotic performance.” This just doesn’t seem true looking at the aggregate results. For the zero-shot (environment steps = 0?) it doesn’t seem to outperform any method. For the xArm in simulation it does worse than TD-MPC and TD-MPC (+offline). It seems to do somewhat better on the D4RL tasks, but from appendix A, it looks like the Hopper is doing most of the heavy lifting. The real world results are good to see.\n\nA significant claim was the uncertainty penalty could be used as a regularizer to improve actions at test-time. I feel like I’m missing something, the results only seem to report what is happening during training (offline then online). Where are the test-time results showing this improvement? Q3.3 shows the effect of parameter λ, but this doesn’t seem to be at test time. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the use of offline data to pretrain a world model that is then fine-tuned with online data, e.g. from a real robot. It uses epistemic uncertainty of the model as a regularizer for online data collection. The methods' effectiveness was demonstrated on simulated and real robots.

The reviewers assessed your submission overall positively, such that I am happy to suggest acceptance. 

Please make sure that the improvement and suggestions that come up in the reviews and the rebuttal are appropriately incorporated in the final version of the paper.
Decision: Accept (Oral)