MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning
Authors: Rafael Rafailov
Kyle Beltran Hatch
Victor Kolev
John D Martin
Mariano Phielipp
Chelsea Finn
Venue: CoRL 2023 Poster (2023)
Keywords: offline RL
online fine-tuning
model-learning
robot learning

Abstract: We study the problem of offline pre-training and online fine-tuning for reinforcement learning from high-dimensional observations in the context of realistic robot tasks. Recent offline model-free approaches successfully use online fine-tuning to either improve the performance of the agent over the data collection policy or adapt to novel tasks. At the same time, model-based RL algorithms have achieved significant progress in sample efficiency and the complexity of the tasks they can solve, yet remain under-utilized in the fine-tuning setting. In this work, we argue that existing methods for high-dimensional model-based offline RL are not suitable for offline-to-online fine-tuning due to issues with distribution shifts, off-dynamics data, and non-stationary rewards. We propose an on-policy model-based method that can efficiently reuse prior data through model-based value expansion and policy regularization, while preventing model exploitation by controlling epistemic uncertainty. We find that our approach successfully solves tasks from the MetaWorld benchmark, as well as the Franka Kitchen robot manipulation environment completely from images. To our knowledge, MOTO is the first and only method to solve this environment from pixels.

MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning
Rafael R
Kyle Hatch*t
Victor Kolev*
John D. Martin
Mariano Phiclipp
Chelsea Finn
Stanforr University Intel Al Labs (rafailov,kharl @css stanford.
Abstract: We study the problem offline pre-training and online fine-tuning for reinforcement learning from high-dimensional observations in the context of re- alistic robot tasks. Recent offline model-free approaches successfully use online fine-tuning either improve the performance of the agent over the data collec- tion policy or adapt to novel tasks. At the same time, model-based RL algorithms have achieved significant progress in sample efficiency and the complexity of the tasks they can solve, yet remain under-utilized in the fine-tuning setting. In this work, we argue that existing methods for high-dimensional model-based offline RI. are not suitable for offline-to-online fine-tuning due to issues with distribution shifts, off-dynamics data, and non-stationary rewards. We propose an on-policy model-based method that can efficiently reuse prior data through model-based value expansion and policy regularization, while preventing model exploitation by controlling epistemic uncertainty. We find that our approach successfully solves tasks from the MetaWorld benchmark, as well as the Franka Kitchen robot manip- ulation environment completely from images. To our knowledge, MOTO is the first and only method solve this environment from pixels.?
Model-based reinforcement learning, offline-to-onlin fine-tuning, high-dimensional observations
Keywords:
1 Introduction
Pre-training and fine-tuning as a paradigm has been instrumental to recent advances in machine learning. In the context of reinforcement learning, this takes the form pre-training a policy with offline learning [1, 2], i.e. when only a static dataset of environment interactions is avail- able; subsequently fine-tuning that policy with limited amount of online fine-tuning. We study the offline-to-onlin fine-tuning problem with focus high-dimensional pixel observations, found in real-world applications, such as robotics.
Prior works for offline-to-onlime fine-tuning often train policy with model-free offline RL objec- tives throughout both the offline and online phases [3, 4, 5, 6, 7, 8). While this approach addresses the challenge of distribution shift in the offline phase, it leads excessive conservatism in the on- line phase, since the policy cannot balance offline conservatism with online exploration. Moreover, model-free works are lacking in generalization abilities, and can even under-perform when trained on non task-specife data [9, 10].
* "Equal contribution
Additional details are available our project wehsite: https://sites /ttp:::///ites..ogglleee. www.gl......veee////o///ooo/ cow/vieu/mo2///
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Offline Pre-training
Online Fine-tuning
(s,â,
Train
T
D
Q a)
world
model
Fine-tuning
D
(s,
agent
Q a)
Train World Model
Train Actor Critic
Figure I: Madel-based offline online fine-tuning. A static dataset of experience is used to train world model, with which the offline actor-critic agent interacts. The actor-critic agent is trained via both data from the environment and data from the model via model-based value expansion. Model data is penalized via an uncertainty penalty which inhibits model exploitation. Finally, during fine-tuning, the agent interacts with the environment, and collects new trajectories, which are used to jointly fine-tune the world model and actor-critic.
Model-based methods, in which the agent learns representation and dynamics model, present an interesting alternative, as they have shown generalization ability to new within-distribution tasks [11, 12, 13, 14]; they are sample-efficient and can enable online exploration via model-generatee rollouts [15]. Lastly, predictive models can also naturally learn stable representations, which makes them suitable for realistic high-dimensional domains [16, 17, 18, 19]. Despite the good case for model-based learning in the offline-to-onling fine-tuning problem, this has been under-explored, with the literature focused mostly on model-free methods.
In this work, we argue that existing algorithms for offline model-based RL are not suitable to the pre-training and fine-tuning or continual learning regimes. In particular, algorithms that use replay buffers model-generadle data, such as [11, 12, 13, 19], create significant distributional shift issues, as the learned model dynamics and reward functions change with additional online interactions. Moreover, models with high-ding ditensiinsii observations, such as [19, 12], deal with the additional complexity of representation shift of the latent data. These algorithms are also not feasible in large models with nigh-dimensional representation spaces, which are common in real-world applications [16, 17, 18]. On the other hand, on- on-policy model-based RL methods such as [20, 21] are amenable to fine-tuning but do not make efficient use of high-quality data in the policy training objective or are not scalable to models with changing representation spaces.
To alleviate these issues, we propose the MOTO (Model-based Offiie-To--Oniinn algorithm. MOTO is model-based actor-critic algorithm which operates in high-dimensional observation spaces. Crucially, MOTO uses model-based value expansion, which removes the need for large replay buffers, alleviates the distributional shift issue, and allows for the use of large-scale predic- tive models while still allowing us to use high-quality offline data in the critic learning. To prevent model exploitation, we additionally implement ensemble model-based uncertainty estimation and policy regularization. We evaluate MOTO on 10 tasks from the MeraW benchmark [22] and two tasks in the Franka Kitchen domain [23, 24], completely from vision. Our approach outperforms baselines 9/10 environments in the Meta World benchmark and solves both settings in the Franka kitchen. As far sion. Moreover, by studying the fine-tuning regime, we empirically validate theoretical performance bounds from prior model-based offline RL.
aware, is the first method to solve this environment completely from vi-
We summarize our contributions as follows: (1) we propose a new model-based actor-critic algo- rithm for offline pre-training and online fine-tuning; (2) we show the first successful solution the Franka Kitchen task from images; (3) empirically verify a proposed theoretical performance gap; (4) to facilitate further research in this area, we will publicly release our environments and datasets.

---

2
Related Work
Our work is at the intersection of offline RL, model-based RL and control from high-dimensional observations (i.e. images). We review related work from these fields below.
Model-Based Offliie RL Model-based offline RL algorithms [20, 11. 25, 21, 26, 19, 12] learn predictive model from the offline dataset and use it for policy training. We would like to design a model-basee reinforcement learning algorithm that can efficiently utilize offline datasets, while being easily amenable to continual learning and online fine-tuning. line of prior works [11, 12, 13] uses MBPO-style optimization [27], which mixes real and model-generated data in a replay buffer used for policy training. [19] generalizes this approach to more realistic domains using variational models and latent ensembles and manages to solve a real robot task involving desk manipulation. However, these methods are not well-suited the tieettnnng tasks, since the data in the replay buffer is sampled from the model's internal representation space, which suffers from significant distribution shift as the model is fine-tuned. Moreover, the need for replay buffers limits the scalability of these algorithms, as state-of-the-art predictive models in many realistic applications (such as autonomous driving [16, 17, 18I) require very large model and representation sizes. Several algorithms such as MOREL [20] and BREMEN [21] use on-policy training within the learned model without the need for large replay buffers, making them well-suited continual learning settings, but cannot use potentially high-quality data from the offline dataset to supervise the actor-critic training.
Variational Dynamics Models Variational predictive models have demonstrated success in a va- riety of challenging applications. One line of research [16, 17, 28, 29, 30] utilizes the model for representation purposes only and uses standard RL, control, or imitation on top it. Others such as [31, 32, 15, 33] use the latent dynamics model either learn policy within the model or deploy shooting-based planning methods. However, most of those prior works focus the online setting and do not make good use of highly-structuree prior data or account for distribution shift. Our method utilizes model-basee value expansion, which allows us to take advantage of the efficiency of model-based training, while also using high-quality offline data for critic supervision.
3 Preliminaries
In this section we review the modeling framework for our world model and epistemic uncertainty estimates.
World Model To model the high-dimensional observations of the environment, VAE based on the RSSM model [31, 32). The model consists of the following components:
recurrent
2t ~ qo (z+ h1, ar.)
latent representation encoder
h, Solz, h ac-1)
deterministic latent state
21 pbl 2t hi)
stochastic latent state
a ~ po [x Ze, hi)
observation decoder
F, (r Ze, ht)
reward decoder
where æ, are the high-dimenssonal environment observations, a, the actions, 7x are the rewards, ht are deterministic latent states, and are stochastic latent states. We denote the latent state [he, 2t] as the concatenation of both. All components of the model are trained jointy via the
ELBO loss as:
Cmodel
E. In po( |s.) In po(pr st)+
Σ t
DEz[go 8, ( 8t-1, am-1)llr (11( a[1)]
(1)
In experiments discrete latent state models, following the Dreamer V2 architecture [15]. Notice that train ensemble of stochastic latent dynamics models (2)))11))))))) following
3

---

Algorithm MOTO: Model-based Offline to Online Fine-tuning
Require: Offline dataset D. initialized policy we and critics Qv, initialized prediction and reward model Mo. policy rollout length H. number of offline training steps Nottine. number of online episodes collect ausieee qpiones.. number of online gradient updates per episode G.
1: II offline pre-training
2: for == 2,2,3, -,...,, Nottine do
3: Sample batch of trajectorics B D.
Updatc Ms on B according Eq.
Generate Htstep latent policy rollouts with penalized rewards (Eq. 3).
6: Update Fe according to Eq. 10
7: update Qe according to Eq. the real and model data.
8: end for
// online fine-tuning
10: for = 1, 2, 3. 3..... .Nonline opisedes do
11: Rollout the policy No in the environment for an episode to collect a new trajectory
12:
D == D UUT
13:
for step 2, ................................ do
Sample a batch of trajectories Update Ms on B according to Eq. 1.
D.
15:
16:
Generate H-step latent policy rollouts with penalized rewards (Eq. 3).
17:
Update Ta according Eq. 10
18:
update Qu according Eq. 9 on the real and model data.
19:
end for
20:
end for
[19] by randomly selecting one model p to optimize at each time step of the trajectory in the model in Eq. 1. This makes the ensemble training no more computationally expensive than single model optimization.
Offline Model-Based RL From High-Dimensional Observations To mitigate issues with model exploitation, similar to [19, 11], we train a latent dynamics model ensemble (s)+1()))) and implement model conservatism by penalizing rewards via dynamics model disagreement, which acts as proxy for epistemic uncertainty. We use the penalty:
uo(s;, at) = std( (Tg. (M++1));11)
where 15(z++1) is the logit outputs of the discrete distribution p (- (+44))). Hence, the final reward function is
fa (s., a, st+1) = re( auug(s at)
where a is a trade-off parameter between reward maximization and conservatism.
4 Model-based Offline to Online Fine-tuning (MOTO)
Our model training architecture and objective follow prior works [19, 34], but we significantly change the actor-critic algorithm optimization towards the goal of efficient online fine-tuning from offline data. We build the MOTO policy optimization procedure on three main design choices: 1) model-based value expansion, 2) uncertainty-aware predictive modelling, and 3) behaviour- regularized policy optimization. The full algorithm training outline presented in Algorithm
Variational Model-Based Value Expansion We would like to train policy via model-based training and without latent replay buffers, while making use of both the high-quality offline data and the world model sources of supervision for the actor and critic. To this end, we adapt ideas from the model-based value expansion literature [35, 36. 37, 38). We consider sequences of data of the form - = (æ1.7, a1.7, T.:T). At each agent training step. we infer latent states si.r g0(81: $1.7 warr, d.:r). We then use the true data as starting points for model-generated rollouts, as:
a Tala )
87 po(ssaaj, 85), 14 i>-pe(re (o(3)))
(4)

---

where the rewards are computed according to Eq. 3. Following standard off- off-poolcy learning algo- rithms, we use critics iQ Qua) and and target networks (Qwi, Qua). We can then use our model to estimate Monte-Carlo based policy returns:
K = Σ
Vi * (5)) = min (6). d)).
(
k-
p
+ +7*VV*
Vo= (B)tk))
k=1
And compute the GAB(y, A) estimate:
WT (*) - A) H-
Σ x
Vz"
V
(5)
We denote (8)(8) AVT (8) + (1 V and optimize the actor objective
H-17
EEE Σ (=
V (8))
HT T-D Fesll
This objective essentially estimates the actor return by mixing Monte-Caria based estimates at various horizons. Notice that is fully differentiable function of the policy parameters, by back- propagating through the Q-functions and dynamics model. Also notice that for H 0 this is just the standard actor-critic policy update.
We can similarly use MC return estimates to train the critics. We recompute the critic target values T ($t) for all states similarly Eq. using the target networks (Qua, Qua). The critics are trained on both the model-generated and real data with the sum of two losses:
H-17
mmddel
E
E
Σ
(V*) (8)
HT T-
1-0μ-1
1
Cdata
E
T
(notice that the second equation does not involves dynamics model samples) and the final critic loss the sum of the two:
Chiall
Cmodel
Cdata
Training the critic networks on the available offline data serves as strong supervision when the dataset already contains rollouts with high returns.
Uncertainty-aware Predictive Modelling In order to prevent model exploitation in the offline training, model-based uncertainty estimates via ensemble statistics, similar to [39, 40, 41, 42, 43, 44, 45, 46]. Note that the loss Cihu is computed transitions sampled from the dataset trajectories through the inference model qe and have ground-truth environment rewards. In contrast, the critic loss Cmadel is computed on synthetic states sampled from the model using only uncertainty- Q,, penalized rewards (Eq 2). This explicitly builds conservatism into the critic values by biasing them towards dataset states and actions. We also considered alternative conservative critic optimization [4, 12]. However, these approaches are incompatible with multi-step returns and require the use of latent replay buffer which undesirable. Following prior works [11, 20, 19] we provide theoretical verification for our modelling choices. addition, by studying the online fine-tuning regime, for able provide empirical verification for prior offline MBRL performance
the first time, bounds. Since the current work does not focus theoretical contributions, defer these results to Appendix B.
5

---

assembly-42
coffee-push-v2
box-close-v2
0opr-0pen-v2
hammer-2
Success Rate
60
disassemble-v2
window open-y?
bin picking v2
drawer-opeen open.v2
plate slide-v2
Success Rate
14
93
62
03
Online Fine-Tuning Steps (thousandss
Ours
Dreamerv2
COMBO
LOMPO
COL
10L SAC BC
The success rates across the 10 Mcta World tasks. MOTO matches or outperforms other methods on 9 out of the 10 tasks, demonstrating MOTO's ability to successfully pre-train offline and fine-tuning online variety of manipulation tasks using limited offline data. DreamerV2 is the only other method achieve competitive results on the MorlW tasks. The model baselines achieve low to moderate performance across all tasks.
Figure 2:
Behaviour Prior Policy Regularization Realistic robot learning datasets often consist of narrow data like planner based rollouts human demonstrations, As such, the initial stages of training dynamics models can be quite inaccurate and the agent can benefit from stronger data regulariza- tion for the policy [26, 13, 25, 21]. To avoid additional complexity of modelling the behaviour distribution we follow an approach similar to [47] which deploy a regularization term the form
)[
log ( 8))s
7 H ( +
7 retj (
T
Advantage over trajectory snippet 8 B
for some function f. The authors suggest simple threshold function works well (i.e adding behaviour cloning term snippets with positive advantage), [13] can also be viewed as instance of this approach using exponential weighting. In this work we focus on realistic robot manipulation tasks with sparse rewards, and just threshold trajectories based on whether they achieve the goals in the environment. We then optimize the joint actor loss:
L
mmodel n
(10)
where is hyper-parameete that wades-off between model based optimization and data regulariza- tion.
Experiments and Results
We aim to answer the following questions: (1) Can MOTO pre-train offline and successfully fine- tune online? (2) What is the impact of different model components? (3) Does MOTO exhibit good generalization and sample efficiency?
Experiment Setup
We evaluate our method two challenging dexterous manipulation domains, MetaWorld [22] and the Standard Franka Kitchen environment [24, 23] used in the D5RL benchmark [48].
MeraWorld contains a variety of simulated manipulation tasks shared, table-top environment be solved with Sawyer robotic arm. We use ten of these tasks for our experiments (see Figure 7). We modify these environments to 64x64 RGB image observations, without any robot pro- prioception and sparse rewards based on task completion. For each environment collected small dataset of 9- 9-10 demonstration episodes using scripted policy.

---

Mixed Task
Partial
Learning Mixed
Learning Curoes Partial
1.0
2.2
1.2
0.5
0.4
0.2
0.2
0.0
0.0
200
300
403
700
200 Votal SC. Unc
300
430
500
Strpo
Disamer1 00030 L0P10 000
NCC
PLi Medel
No UnC.
Dreamervz
Figure 3: (Left) Success rate of completing the "mixed" and "partial" tasks in Franka Kitchen. MOTO outper- forms all methods on both tasks, and is the only method to achieve meaningful progress on the "partial" task, indicating MOTO's capacity for combinatorial generalization. (Right) We carry out ablations design: no uncertainty penalties "No Unc.", no behavioral cloning regularization "No. BC", and removing both "No BC., No Unc."; removing model-based value expansion as well gives us DreamerV2. We observe that the gains from each component are additive, and only the full model achieves full performance. Lastly, since all ablations share the same architecture. this shows that the performance improvement is not due to stronger architecture, but rather the actor critic training.
the MOTO
We also evaluate MOTO on the Standard Franka Kitchen environment from the DSRL benchmark [48], which is a challenging long-range control problem that requires using simulated 9-DOF Franka Emika Robot to manipulate multiple different objects in simulated kitchen area. For our experiments, we only use the central camera image without the wrist camera view or rohot propri- ocpetion. Since the "partial" task does not contain successful trajectories for all four target objects, we only regularize policy training with respect to the first three objects.
More details about the environments and datasets can be found in Appendix C.
Baselines We compare our method to prior vision-based offline model-based RL algorithms LOMPO [19] and COMBO [12] as well as DreamerV2 [15], statc-of-the art online model-based learning algorithm. We also compare our approach against CQL., [4] successful model-free offline RL algorithm, IQL, [3] a state-of-the art model-free regression-
fine-tuning algorithm, SAC [49], and behaviour cloning. All methods are pre-trained offline for 10 thousand gradient steps and fine-tuned with online interactions for a total of 500 thousand environment steps.
Meta World Results Results for the Meta World tasks are in Fig. 2. MOTO outperforms other methods on 9 out of 10 tasks in that domain. This shows MOTO's ability to successfully pre-train offline and fine-tune online on a variety of manipulation tasks using limited offline data. Dream- erV2 is the only other method achieve competitive results on the MetaWorld tasks. The model free baselines achieve low moderate performance across all tasks. Perhaps somewhat surpris- ingly, COMBO and LOMPO achieve very low success rates most of the tasks. A speculative explanation for this is that the MetaWorld environments have a significant degree of randomization between them at each new episode, causing the learned image representations to change frequently. Since COMBO and LOMPO are off-policy methods that maintain replay buffers of the latent state representations, if these representations change frequently this would lead to poor performance.
Franka Kitchen Results As seen in Fig. 3, our method successfully solves both the "mixed" and "partial" tasks of Franka Kitchen with 100% and 90.5% final success rates respectively. DreamerV2 the only other method to obtain non-trivial success rates. Noteworthy MOTO's success on the "partial" task, which demonstrates that the world model is capable of combinatoric generalization.
While model-free methods make some progress, ultimately they stagnate and cannot successfully complete all four objects on either task. This is likely due to the partial observability of the envi- ronment, since the robot can occlude manipulated objects and also requires joint state estimation di- rectly from images. In contrast, variational models serve as Bayesian filters and naturally build state estimations of the environment in the latent space. Model-based methods LOMPO and COMBO achieve very limited progress, due to the non-stationarity issues described in the beginning of Sec- tion 4. The DreamerV2 algorithm learns more slowly and only reaches final success rates of 77.5%
7

---

and 13.5% versus 100% and 90.5% for our method on the "mixed" and "partial" task. As far as we know, our model is the first method to solve the Franka Kitchen environment from images.
Ablation Studies In this section we evaluate the contribution of each model component to final performance. Results are presented in Fig. (right). We also include the standard DreamerV2 algorithm for direct model-based comparison. While all ablations make significant progress on the "mixed" task, only the full model manages solve it entirely. The full model outperforms the others on the "partial" task by very significant margin. We also note that without any behavioral cloning (BC) data regularization, both the "No BC., No Une." and DreamerV2 methods learn unsafe behaviours, such as hitting the kettle into the goal position, or smashing the light switch with the robot head, instead of using its gripper to grasp and place the objects. These policies would be unsafe for both the hardware and environment any of the regularized methods (videos are available on the project website).
real setting. Such behaviours are not present in
Discussion
Future Work The MOTO algorithm design does not require large replay buffers of intermediate representations, while still allowing the use of high-quality data to supervise the critic learning and bootstrap the policy optimization. We believe that these qualities make MOTO very suitable for realistic applications, which require large scale models [16, 17, 18]. We plan to evaluate MOTO on large-scale realistic domains, such as CARLA [50] in future work.
MOTO is also well-suited to the model-based imitation learning setting [51, 14, 52, 53], which has recently been successfully applied real world scenarios as well [54, 55J. By using on-policy roll- outs, MOTO can maintain the stability and theoretical guarantees of adversarial imitation learning [56, 57, 19]. while still using the high-quality expert data to both provide supervision to the critic, as well as to regularize the policy.
Limitations MOTO builds a level of pessimism through penalizing state-action epistemic model uncertainty. Excessive pessimism can prevent the model from exploring or generalizing outside of the available data distribution. This can hurt performance if the offline dataset consists of lower- quality incomplete data. MOTO also uses policy regularization, which is based task success. It may be difficult to adapt this approach to tasks that involve more complex non-sparse rewards. Finally, key component of MOTO is controlling model-based epistemic uncertainty. We train an ensemble of latent transition models and use their disagreement as a reward penalty. The models we consider use MLPs for the latent dynamics, however it is not clear if this scheme can transfer to more complex architectures, such as Transformers, which are now more widely used within the predictive modeling context.
7 Conclusion
We present MOTO, a model-based reinforcement learning algorithm specifically designed for the offline pre-training and down-stream fine-tuning regime. MOTO learns variational model directly from pixels, and trains an actor-critic agent within the learned latent dynamics model using model- based value expansion, epistemic uncertainty corrections, and policy regularization, all of which we show have an impact in improving performance, and deriving safe and robust policies. MOTO outperforms baselines in terms of sample efficiency and final performance on 9/10 MetaWorld tasks and far as we are aware is the first and only method to solve the Franka Kitchen benchmark from images. Furthermore, by studying the offline pre-training and fine-tuning regime, we empirically verify long-standing theoretical results on the offline model- RL problem. Finally, the structure of the algorithm makes it suitable for use in very large scale dynamics models, such as the ones used in autonomous driving, which prior work not able to utilize, well as backbone for model- based imitation, multi-task and transfer learning. We plan to explore these directions in follow-up works.
8

---

Acknowledgments
We would like to express gratitude the reviewers, whose feedback helped improve this paper. Chelsea Finn is CIFAR Fellow in the Learning in Machines and Brains program. This work was supported by Intel Al Labs and ONR grant N00014-21-1-2685.
References
[1] S. Lange, T. Gabel, and M. A. Riedmiller. Batch reinforcement learning. In Reinforcement Learning, volume 12. Springer, 2012.
[2] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005..06643, 2020.
[3] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06100 2021.
[4] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning. arXiv preprint arXiv:2006.04779 2020.
[5] A. Nair, M. Dalal, A. Gupta, and S. Levine. Accelerating online reinforcement learning with offline datasets, arXiv preprint arXiv:2006.092: 09359, 2020.
[6] M. Yang and O. Nachum. Representation matters: offline pretraining for sequential decision making. In International Conference on Machine Learning, pages 11784-11794, PMLR, 2021.
[7] I.. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34:15084-15097, 15097, 2021.
[8] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, Kay, J. T. Springenberg, et al. generalist agent. arXiv preprint arXiv:2205.0617 06105, 2022.
[9] T. Yu, A. Kumar, Y. Chebotar, K. Hausman, S. Levine, and C. Finn. Conservative data shar- ing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems, 34:1 1501-11516, 2021.
[10] T. Yu, A. Kumar, Y. Chebotar, C. Finn, S. Levine, and K. Hausman. Data sharing without rewards multi-task offline reinforcement learning. 2021.
[11] T. Yu, G. Thomas, L. Yu, S. Ermon, Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv preprint htt.::2005//33333 2020.
[12] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn. Combo: Conservative offline model-based policy optimization. Advances neural information processing systems, 34:28954-28967, 2021.
[13] C. Cang, A. Rajeswaran, Abbeel, and M. Laskin. Behavioral priors and dynamics models: Improving performance and domain transfer in offline rl. arXiv preprint arXiv:2106.09119 2021.
[14] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Visual adversarial imitation learning using variational models. Advances in Neural Information Processing Systems, 24:3016-3028,, 2021.
[15] D. Hafner, T. Lillicrap. M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprint arXh::2010.02193, 2020.
[16] A. Hu, G. Corrado, N. Griffiths, Z. Murez, C. Gurau, H. Yeo, A. Kendall, R. Cipolla, and J. Shotton. Model-based imitation learning for urban driving. arXiv preprint
arXiv:2210.07729, 2022.

---

[17] A. Hu, Z. Murez, N. Mohan, S. Dudas, J. Hawke, V. Badrinarayanan, R. Cipolla, and A. Kendall. Fiery: Future instance prediction in b-rd view from surround monocular cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15273-15282, 2021.
[18] A. K. Akan and F. Güney, Stretchbev: Stretching future instance prediction spatially and temporally. arXiv preprini arXiv:2203. 13641, 2022.
[19] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Offline reinforcement learning from images with latent space models. ArXiv, abs/2012.11547. 2020.
[20] R. Kidambi, A. Rajeswaran, P. Netrapalli, and Joachims. Morel: Model-based offline rein- forcement learning. arXiv preprint I 2020.
[21] T. Matsushima, H. Furuta, Y. Matsuo, O. Nachum, and S. Gu. Deployment-effccien reinfor reinforce- ment learning via model-based offline optimization. arXiv preprint arXiv:2006.03647, 2020.
[22] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pages 1094-1100. PMLR, 2020.
[23] A. Gupta, Kumar, C. Lyneh, Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv: 1956, 2019.
[24] Fu, A. Kumar, O. Nachum, Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint wxXiv:2004.02//11 2020.
[25] A. Argenson and G. Dulac-Amold. Model-based offline planning. arXiv preprint arXiv:2008.05556, 2020.
[26] P. Swazinna, S. Udluft, and T. Runkler. Overcoming model bias for robust offline deep rein- forcement learning, arXiv preprint arXiv:2008. 2xXiv:0008.05333, 2020.
[27] M. Janner, J. Fu, M. Zhang, and Levine. When trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, pages 12498-12509, 2019.
[28] M. Watter, Springenberg. J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems, pages
2015.
[29] M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. Johnson, and S. Levine. Solar: Deep struc- tured representations for model-based reinforcement learning. In International Conference on Machine Learning, pages 7444-7453. PMLR, 2019.
[30] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critie: Deep rein- forcement learning with a latent variable model. arXiv preprint help:1907.0053,,,0 2020.
[31] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. International Conference Learning Representations, 2019.
[32] D. Hafner, T. Lillicrap, Ba, and M. Norouzi. Dream control: Learning behaviors by latent imagination. International Conference Learning Representations, 2020.
[33] D. Ha and J. Schmidhuber. World models. arXiv preprint 1803. /0/22, 2018.
[34] R. Sekar, O. Rybkin, K. Daniilidis, P. Abbeel, D. Hafner, and D. Pathak. Planning explore via sclf-supervised world models. arXiv preprint arXiv:2005.0540 055600 2020.
10

---

[35] V. Feinberg, A. Wan, I. Stoica, M. I. Jordan, J. E. Gonzalez, and S. Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv: 1803.0010,, 2018.
[36] J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. Advances in neural information processing systems, 31, 2018.
[37] B. Amos, S. Stanton, D. Yarats, and A. G. Wilson. On the model-based stochastic value gradient for continuous reinforcement learning. In Learning for Dynamics and Control, pages 6-20. PMLR, 2021.
[38] I. Clavera, V. Fu, and P. Abbeel. Model-augmented actor-critic: Backpropagating through paths. arXiv preprint urXiv:2005.0800 2020.
[39] K. Chua, R. Calandra, R. McAllister, and Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018
[40] I. Clavera, J. Rothfuss, J. Schulman, Y. Fujita, T. Asfour, and Abbeel. Model-based rein- forcement learning via meta-policy optimization. In Conference on Robot Learning. pages 617-629. PMLR, 2018.
[41] M. Deisenroth and C. E. Rasmussen, Pilco: A model-based and data-efficient approach to pol- icy search. In Proceedings of the 28th International Conference on machine learning (ICML- 11), pages 465-472, 2011.
[42] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv: 10592, 2018.
[43] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMLR, 2020.
[44] Y. Luo, H. Xu, Y. Li, Y. Tian, T. Darrell, and T. Ma. Algorithmic framework for model-base deep reinforcement learning with theoretical guarantees. arXiv preprint arXiv: /807.0388,, 2018.
[45] A. L.. Strehl and M. L.. Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 730991130911331 2008
[46] A. Zanette and Brunskill. Tighter problem-dependent regret bounds in reinforcement learn- ing without domain knowledge using value function bounds. In International Conference on Machine Learning, pages 7304-7312. PMLR, 2019.
[47] N. Y. Siegel, T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, N. Heess, and M. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning, 2020.
[48] R. Rafailov, K. B. Hatch, A. Singh, A. Kumnr, L. Smith, I. Kostrikov, P. Hansen- Estrucc, V. Kolev, P. J. Ball, J. Wu, S. Levine, and C. Finn. D5rl: Diverse datasets for data-driven deep reinforcement learning, 2023.
[49] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint 1rXiv:1801.01200 2018.
[50] A. Dosovitskiy, Ros, F. Codevilla, A. Lopez, and V. Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning. pages 1-16, 2017.
[51] N. Baram, O. Anschel, I. Caspi, and S. Mannor. End-to-end differentiable adversarial imitation learning. In International Conference Machine Learning, pages 390-399. PMLR, 2017.

---

[52] J. D. Chang, M. Uehara, D. Sreenivas, R. Kidambi, and W. Sun. Mitigating covariate shift in imitation learning via offline data without great coverage. arXiv preprint arXiv:2106.03222 2021.
[53] W. Zhang, H. Xu, H. Niu, P. Cheng, M. Li, H. Zhang, G. Zhou, and X. Zhan. Discriminator- guided model-bases offline imitation learning. arXiv preprint arii:2207.002444 2022.
[54] Y. Lu, Fu, G. Tucker, X. Pan, E. Bronstein, B. Roelofs, B. Sapp. B. White, Faust, White- son, et al. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios, arXiv preprint arXiv:2212.114119 2022.
[55] E. Bronstein, M. Palatucei, D. Notz, B. White, A. Kuefler, Lu, S. Paul, Nikdel, Mougin, H. Chen, et al. Hierarchical model-based imitation learning for planning autonomous driv- ing. 2022 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 8652-8653 IEEE, 2022.
[56] Ho and Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[57] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 512-519, IEEE, 2016.
[58] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555-2565 PMLR, 2019.
[59] I. Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 10 2022. URL https: httpp://githhut wwt.coooo/oiaa. comm/Koottrikoo/aarr22 v2.
[60] G. Barth-Maron, M. Hoffman, D. Budden, W. Dabney, D. Horgan, D. Tb, Muldal, N. Heess, and T. Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint 1804vv1804.006//1 08617, 2018.
12

---

A Additional Experiments
A.1 Model-Based Generalization
The "partial" task also provides good test bed for an algorithm's generalization capabilities, since the offline dataset does not contain full solutions for it. This is different problem than the standard dynamic programming ("stitching") issue of data- centric reinforcement learning since the dataset does not contain a sequence of state-action pairs that lead from the initial state to the goal state. Instead, to solve this task, a learning agent must understand the compositional nature of the scene and do com- binatorial generalization
Agent Expected
Colidde Stap
Rotictt Step
the objects. In this section we seek to answer whether 1) the learned model can do combinatorial generalization of within distribution tasks and 2) whether policy optimiza- tion can take advantage of the model's capabilities. We evaluate the agent the end of the offline pre- training phase. To answer the first question, we con- sider episodes that successfully complete the "par- tial" task from the trained agent. We condition model on the frames that solves the first three tasks (which are covered in the offline dataset) and rollout the expert actions to predict the following frames. Results are shown in Fig. 1. The model successfully predicts a combination the mi- crowave, kettle, bottom burner and light switch in the correct configuration, despite never encoun- tering these four objects together in the offline dataset. Moreover, we evaluate the model-predicted rewards on these expert trajectories, plotted in Fig. 4 (left). We see that the model predicts rewards of up to 4 with average reward of 3.63, despite only being trained on trajectories with maximum reward of 3. This results show that the learned model is capable of compositional generalization. To evaluate whether the learned policy can take advantage of the model generalization capabilities, we rollout the trained agent under the model and evaluate the predicted rewards, results are shown in Fig. 4 (right). The agent achieves an average final reward of 3.52 under the learned model and solves all four tasks. This suggest that the model- based RL agent is able to combinatorial generalization, but the offline dataset is not enough adequately learn the environment dynamics.
Figure 4: We evaluate the model's gener- alization capabilities at the end of the of- fline pre-training phase. The model correctly predicts rewards up to 4 on successful episodes in the "partial" task, even though the maximum dataset reward is 3. (left).
When doing rollouts in the learned model, the policy solves all four objects in the "par- tial" task and reaches rewards of up to (right).
A.2 Constrained Offline Data Ablation
We aim to test the performance of MOTO under a data- constrained regime, evaluating whether learning slows down with less offline data. To do so, we randomply sampled 100 and 250 episodes from the Franka Kitchen dataset and used them for offline training, evaluating on the "Mixed" task. The results are presented in fig. We observe that learning not slowed down by reduction in offline data, at least to the extent that we tested. This shows that MOTO is robust to a constrained set of offline data, and can operate at the same performance level even with 5 times less offline episodes. Notably, we hypothe- sise there minimum threshold for diversity offline learning, yet we see no performance degredation even at 100 episodes. It is important point out that episodes were randomly sampled without regards to the reward at-
Mard
Teal commen 201011 2huuard
Figure 5: Training curves for data ab- lation experiments. We see no degra- dation performance when using only 100 and 250 pre-training episodes.
13

---

tained in each episode, i.e. the 100 episodes are not of proportionally higher quality.
B Theoretical Results and Empirical Validation
Theoretical Results for -nwer Model- based Training Given our choice of variational parametrization and model uncertainty estimation we can directly adapt certain theoretical guarantees from prior model-based RL literature [11, 20, 19]. We consider the following result in particular: let To(s'|s, a) and T(*'s,, a) be the learned and true latent dynamics models respectively. We define the discounted state-action distri- bution
Empirical Uncertainty Bound
4
Model Unc. (Normalized)
Performance Gap
3
Steps (thousands)
PT No (8, a)x
Σ P T.o (8: =8) (a((s)
t=0
in the standard way. The function u(s, a) is an admissible error estimator
Figure 6: Empirical evaluation of The- orem B.I. We plot the performance gap versus the the empirical estimates of (normalized) expected model uncer- tainty using Eq. 12.
d=[T(s' a) (T (s, a)] & u(s,a) a).
For any policy ı we can then define
ca(TT) ((a [u(s,a)]
The following Theorem then holds:
Theorem B.1. (Informal) Let (*))) be the optimal policy under the learned model To(s'(s, a) with an uncertainuy-penaliee reward and n* the optimal policy in the ground- MDP. Under certain mild assumptions, then the following inequality holds:
20eu(T**) 3 Ex
Σ
r2
Eas
Σ Tr
(1I)
t=0
Proof. Consult [11].
Empirical verification From the Theorem, we can deduce that the policy under-performen is upper bounded by the discounted model-based uncertainty over the state-action distribution induced by the expert policy under the learned model. In practice we do not have access to an oracle esti- mator u(s, a) and we use the ensemble disagreement from Eq. While these results are not new, empirical verification difficult in the fully offline case, since we have static dataset, and all values point estimates. However, in the online fine-tuning case, we have continuum of datasets and can empirically verify the claims of Theorem B.
We periodically evaluate c.(TT) (T*" and the expected model uncertainty induced under the expert state- action distribution in the learned model. At each epoch E, we cannot generate model rollouts from the expert, since that would require training an expert policy under the current inference model goz- However, we can sample expert episodes from the trained expert and the environment. Given an expert trajectory rxp T1.7; a1.7 sample latent belief states from the first T H steps to obtain $1:(T-M) ~ go (- (1:T-H). From each state 8, we then rollout the expert actions ajs+H using the current iteration of the dynamics model Tég and obtain states (($), " as in Section 4 (here a² aj+z from the expert dataset. We can then obtain the empirical estimate of
cu(
& E Egg (6% T
Σ uo a²)]
(12)
H(T - H)
Empirical results evaluated on the "partial" task shown in Fig. 6. We that the performance gap is strongly bounded (up choice of the penalty scale) by the estimate from Eq. 12, which verifies the claim of Theorem B.I.
14

---

Visualization of the different MetaWorld enviromments used in experiments. row from left right: assembly-v2, bin-picking-ve box- close-22, -rffee--ush-v2, disassemble-v2. Bottom row from left to right: door-open-v2. draver-open-ve hammer-v2, plate-slides 2on2,,inndee windov-open-v2
Figure 7:
C Experimental Details
C.1 Environments
The Franka Kitchen environment from [23] (RPL) is a challenging long-range control problem, which involves simulated 9-DOF Franka Emika Robot in a kitchen setting. The robot uses joint- space control and the observation a single 64x64 RGB image; we not assume access object states or robot proprioception. The goal of the agent manipulate set of 4 pre-defined objects and receives a reward of 1.0 for each object in right configuration at each time step. This is very challenging environment due 1) high- ditenmiimiii observation spaces; 2) partial observability with non-trivial object and robot state estimation; 3) need for very-fineggraiie control in order to operate the small elements of the environment, such as turning knobs and flipping the light-switch; 4) the long-range nature of the tasks; 5) the use of sparse rewards, which provide limited intermediate supervision to the policy, and finally 6) the use of high-dimensional control which requires learning forward kinematics from images alone. For our experiments we render the original RPL datasets and consider two environments from the D4RL benchmark [24]. The "mixed" task requires operating the microwave, kettle, light switch and slide cabinet and has small set of successful demos in the offline dataset. The "partial" task, which requires manipulating the microwave, kettle, bottom burner and light switch does not have any trajectories that successfully complete all four objects, but has demonstrations for several configurations which complete to three objects. We will release this dataset with our project to facilitate the development and testing of vision-based offline RL algorithms.
For the model-free methods, since framestack of 3 for all of our model-free experiments. At each timestep f, the agent was provided with a history of the previous 3 images (from the offline trajectories during offline training, or from the environment during online training). For COMBO and LOMPO, since the latent dynamics model has recurrent component and therefore can implicitly retain history of observations, we did not use any framestacking with the image observations from the environments.
feedforward network for encoding images,
One the Franka Kitchen Environment, we did not use an action repeat, and on the Metaworld en- vironments and data we used an action repeat of 2. For the online finetuning experiments, we used the following procedure: roll out the current policy in the environment for a single episode, add that episode to the replay buffer, and then finetuning the model, critic network, and the policy network. the Franka Kitchen environment, after each episode we performed 50 gradient steps on each component of each method (eg: model, critic network, and the policy network). For the Metaworld environments, performed 20 gradient steps after each episode. In total, the Franka Kitchen environments, we performed 10, 000 gradient steps of offline training and 66, 300gradient steps of
15

---

Environment
Avg. Return
Success Rate
assembly-v2
36.000
1.000
bin-picking-v2
20.900
1.000
box-close-v2
25.300
1.000
coffee-push-v2
36.200
1.000
disassemble-v2
31.556
1.000
door-open-v2
15.200
1.000
drawer-open-v2
48.000
1.000
hammer-v2
63.333
1.000
plate-slide-v2
71.100
1.000
window-open-v2
60.500
1.000
Table 1: Undiscounted episode returns and success rates in the MetaWorld datasets.
online finetuning- On the Metaworld environments, we performed 1, 000 gradient steps of offline training and 20, 000 gradient steps of online finetuning.
C.2
Datasets
Kitchen
Number of trajectories: 563
Number of transitions: 128, 569
Average undiscounted episode return: 261.12
Average number objects manipulated per episode: 3.98
MetaWorld All of the MetaWorld datasets have 9 10 trajectories and 1, 010 total transitions. The average undiscounted episode returns and success rates are shown in Table
C.3 Model Based Methods
MOTO uses the model architecture from [32]. For the convolutional image encoder network, the following hyperparameters:
channels: (48, 96, 192, 384)
kernel sizes: (4, 4, 4,
strides: 2,,
padding: VALID
four final MLP layers of size: 400
The decoder network consists of Deconolution/Transpoe convolution layers with the following hyperparameters:
four initial MLP layers of size: 400
channels: (128, 64, 32, 3)
kernel sizes: (5, 5, 6, 6)
strides: (2, 2,
padding: VALID
16

---

MOTO was trained using model learning rate of 1 x 10 44 The critic and policy network learning rates are 8 x 10-5. The batch size for model training is 16 and the batch size for agent training is 128. We also used a filtered behavioral cloning factor of 10 and a disagreement penalty factor of 10.
The latent dynamics model is represented using RSSM [58] with ensemble size of models. All other hyperparameters are the default values in the DreamerV2 repository.
The V2 baseline uses the same hyperparameters as used for MOTO (excluding the behav- ioral cloning factor and the disagreement penalty factor).
COMBO [12] and LOMPO [19] repository. For the image encoder network of the model, use the default convolutional encoder architecture, which has the following hyperparameters:
using the image- maage-hased implementations from the LOMPO
channels: (32, 64, 128, 256)
kernel sizes: (4, 4, 4, 4)
strides: (2, 2, 2, 2)
padding: VALID
final MLP layer size: 1024
Similarly, the decoder network consists Deconvolution/TTanspoa convolution layers with the following hyperparameters:
initial MLP layer size: 1024
channels: (128, 64, 32, 3)
kernel sizes: (5, 5, 6, 6)
strides: 2,,
padding: VALID
The latent dynamics model is represented using an RSSM [58] with an ensemble size of models.
Both COMBO and LOMPO were trained using a model learning rate 66x 10 and critic network learning rate of x 10-4 and policy network learning rate of 3 x 10-4 The batch size for model training is 64 and the batch size for agent training is 256. For COMBO,
conservatism penalty factor of 2.5, and for LOMPO we use a disagreement penalty factor of X 5.
C.4 Model Free Methods
The model free baselines (IQL [3], CQL [4], SAC [49], BC) were run using the JAXRL2 frame- work [59]. For all policy networks, critic networks, and value networks, we used a feed-forward convolutional encoder network architecture from the D4PG method [60], with the following hyper- parameters:
channels: (32, 64, 128, 256)
kernel sizes: (3, 3, 3, 3)
strides: (2, 2, 2, 2)
padding: VALID
final MLP layer size: 50
This encoder was then followed by two MLP layers of size 256, followed by a final output layer of size 1 (for critic and value networks) or of size action-dim for policy networks. ReLU activations were used between each layer.
We use discount factor 0.99 and batch size of 256 for all of the methods, well learning rate of 3 1 for all policy, critic, and value networks. We also used soft target update
17

---

for critic and value networks with factor of r = 0.005. For CQL we set the conservatism penalty factor a = 5, and for IQL. we set the expectile hyperparameter 7 = 0.5 and the inverse temperature hyperparameter 3 = 3, which are the default values in JAXRL2. For all other hyperparameters, we used the default values in JAXRL2.
18


Reviews:
[{"review_id": "yNc4uotCY2b", "reviewer": "Reviewer_Ar1n", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors consider the problem of online fine-tuning model-based RL approaches that were pre-trained on offline data. They argue that existing methods do not solve this problem sufficiently, as offline model-based RL algorithms fail at fine-tuning. The proposed solution relies on introducing model-based value expansion as a way to train the policy and the value networks and properly regularizing the policy during fine-tuning. The authors show that using this approach they can outperform prior work on the Meta-World suite as well as the challenging Franka Kitchen tasks. Finally, the authors present ablation studies and discuss the limitations of their study.", "summary_of_recommendation": "The quality of the paper is quite high, but there are some important issues that I think should be fixed before publishing the paper. As such, for now, I decided to go with weak reject. I would be happy to increase the score after addressing these issues.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See the weaknesses section above. In particular, I would appreciate:\n- A more thorough discussion of the value expansion technique.\n- An investigation of the impact of the architecture\n- A discussion of the computational costs of the method.\n- Fixing the minor presentation errors.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The problem of the adaptation of model-based RL algorithms is quite an important one, and going from the offline to the online setting has been relatively understudied. As such, this paper deals with a crucial issue in RL and could potentially have a high impact.\n- The method proposed by the authors shows strong empirical results, especially in terms of generalization capabilities on Franka Kitchen.\n- Related work is nicely described.\n\nWeaknesses:\n- The value expansion seems to be the central contribution proposed in the paper, but its motivation is not sufficiently discussed and its impact is not properly examined. Why is it that value expansion would help with the replay problem? How is your approach different from the value expansion in Dreamer-v2 you base on (they also use GAE-like expansion)? How is the horizon length set and how much does it impact the results? I think the paper would benefit from a more detailed discussion and more ablation studies on this part.\n- The impact of the architecture on the results has not been studied. I find this important, since MOTO uses Dreamer-like architecture, while all the baselines use a smaller architecture with a higher-dimensional latent space. Currently, it is not clear whether the improvements offered by Dreamer and MOTO are solely due to the algorithmic changes or also due to a better-suited architecture. \n- The paper does not discuss the computational expenses of the method. Given the complexity of the method, it would be good to have an insight into how expensive it is to train it, e.g. how many GPU hours it took to train it vs. the rest of the baselines.\n- There are no real-world experiments in the paper.\n- Minor point: the presentation of the paper should be improved in some regards:  \n   - Figure 4 mentions  \"No BC\", but this abbreviation is not explained in the text. I assume you mean Behavioral Cloning, i.e. behaviour prior policy regularization?\n   - It's difficult to tell apart colors in Figure 4 (dark blue vs black).\n   - Line 58: \"kicthen domain\" -> kitchen\n   - Line 76: \"is sampled form\" -> \"sampled from\"\n   - Line 111: \"where α is a tarde-off\" -> \"trade-off\"", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "szZ7TkDLJ8g", "reviewer": "Reviewer_kYGq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes MOTO, a new model-based algorithm for the offline pre-training to online fine-tuning regime. MOTO operates on pixel inputs using the DreamerV2 architecture, with adjustments made to the model-based actor-critic algorithm, such as MVE for the actor and critic losses, and uncertainty regularization based on an ensemble of dynamics models. The proposed method shows favorable results compared to baselines on simulated robotic manipulation tasks. ", "summary_of_recommendation": "While the approach presented in this paper may not be easily transferrable to real robots, its impressive results on pixel-based robotic manipulation tasks in simulation, as well as its improvement of the state-of-the-art in model-based RL render it a worthy contribution to the robot learning community.  ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "In the MetaWorld results, seems like most of the advantage of MOTO comes from its DreamerV2 architecture. Have the authors attempted to insert the DreamerV2 architecture into any of the other model-based offline-RL baselines for comparison?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper is well written and states its contributions clearly. The proposed algorithm is a novel combination of existing components, which are well suited to solve the tasks selected for the experiments. \n\nImpressively small dataset of offline trajectories for training in the MetaWorld task.\n\nAdditionally, Impressive results on solving the Franka Kitchen environment from pixels. \n\n\nWhile solving robotic manipulation environments from pixels is a step towards usability of model-based RL on real-world robots, it seems unlikely that the proposed algorithm will be used in a real-world scenario anytime soon, for a few reasons:\n\n- Task completion reward from the simulator may not be readily available in real-world scenarios\n- Real-world visual observations may be much noisier and more diverse than simulated images.\n- The reported 500K environment steps required for training are prohibitive for most applications involving real-world hardware.\n- Another detriment to real-world robot tasks is mentioned in the limitations - MOTO may deteriorate if the offline dataset is lower-quality or incomplete, which is a common trait of real-world datasets.\n\n\nSome tiny nit-picks:\n- Line 111: tarde → trade \n- Broken citation on line 113", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "sN2mHt6yQSw", "reviewer": "Reviewer_KHbi", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes model-based reinforcement learning in the offline pre-training to online fine-tuning setting. During training, it mainly introduces three techniques to achieve good performance: 1. training the Q function on both dataset and unrolled trajectories by the learned dynamics model; 2. add reward penalty by considering the model uncertainty when unrolling the dynamics model; 3. add behavior prior policy regularization when training the policy. The proposed method achieves good performance on pixel-based offline-to-online tasks.", "summary_of_recommendation": "The paper is easy to follow and well-written and each design choice is well-motivated.  The proposed method achieve strong performance on a set of pixel-based tasks. My main concern is that three newly introduced components compared to Dreamer V2 are already used in previous works, especially two of them are known to work in offline reinforcement learning settings, e.g., considering model uncertainty and adding behavior cloning loss. People would expect combining these techniques works in the offline-to-online setting, which limits the significance of this work. Therefore, my recommendation is Weak Reject.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please check the weaknesses in the Review section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The paper is very easy to follow and well-written.\n- The proposed method is clean and each design choice is well-motivated\n- The proposed method achieves good performance on a set of tasks compared to baselines, especially on the Franka Kitchen task from images.\n\nWeaknesses:\n- Although the proposed method works well in the tested tasks, each proposed component is not new and has already used in previous methods, including\n    - The Q function is jointly trained by both model-based loss and model-free loss. This idea is similar to the loss used in, e.g., [1]\n    - Considering uncertainty when unrolling the trained dynamics model. This idea is similar to [2] and several other model-based offline RL papers.\n    - The idea of adding behavior cloning when updating the policy is also used before, e.g., [3][4][5] \n\nAll these methods are known to work in their settings and one would expect the combination of these techniques can work in the offline-to-online setting. I can understand that combining these techniques and demonstrating promising results on a set of tasks take effort, however, the insights obtained from this paper are limited.\n\n\n[1] Ma, Xiao, et al. \"Contrastive variational reinforcement learning for complex observations.\" Conference on Robot Learning. PMLR, 2021.\n[2] Rafailov, Rafael, et al. \"Offline reinforcement learning from images with latent space models.\" Learning for Dynamics and Control. PMLR, 2021.\n[3] Fujimoto, Scott, and Shixiang Shane Gu. \"A minimalist approach to offline reinforcement learning.\" Advances in neural information processing systems 34 (2021): 20132-20145.\n[4] Zhao, Yi, et al. \"Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning.\" arXiv preprint arXiv:2210.13846 (2022).\n[5] Siegel, Noah Y., et al. \"Keep doing what worked: Behavioral modelling priors for offline reinforcement learning.\" arXiv preprint arXiv:2002.08396 (2020).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "X5yPyrOv8CY", "reviewer": "Reviewer_ZAHx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces MOTO (Model-based Offline-To-Online) as a new fine-tuning approach for online reinforcement learning in the context of robot tasks using high-dimensional observations. \n\nThe authors argue that existing high-dimensional, model-based offline reinforcement learning methods aren't suitable for offline-to-online fine-tuning due to issues like distribution shifts, off-dynamics data, and non-stationary rewards.\n\nMOTO is proposed as an on-policy model-based method that can efficiently reuse prior knowledge collected offline, while also being able to adapt online to novel tasks. MOTO has three parts: model-based value expansion, uncertainty-aware latent dynamic model and behavior-regularized policy optimization (adding a weighted imitation learning terms).\n\nThe paper demonstrates that MOTO outperforms recent model-free and model-based methods on several benchmark tasks, helping to improve both performance and sample efficiency.", "summary_of_recommendation": "This work connects established method from offline RL, model-based RL and imitation learning and showcases a working framework toward offline pretraining and online fine-tuning. I recommend acceptance as how to utilize large-scale dataset to facilitate real-world robotics is a very important problem and this work provides a method.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* The figure 1 is in extremely poor quality.\n* In line 113 page 4, these is a missing reference.\n* In line 144, when discussing \"using only uncertainty-penalized reward\" you can refer back to Eq. 2 for clarity.\n* The authors should clarify how the size and quality of the offline dataset affect the performance of MOTO. This would provide more insights into the data efficiency of the method.\n* We can also discuss the \"ratio\" of offline pretraining and online fine-tuning. A question is when increasing the pretraining dataset while decreasing the online interactions, could the model achieve similar result? If not then what is the relationship between the size of two dataset?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Quality**: The quality of this work is good. The paper is well-structured, the methodology is clearly explained, and the results are backed by comprehensive experiments. The paper combines several existing methods to build a working framework with offline pretraining and online fine-tuning setting, where the setting itself is very important.\n\n**Clarity**: The paper is well-written and the clarity is good.\n\n**Originality**: The work is original in its approach to merging offline pre-training and online fine-tuning in a model-based reinforcement learning context. The MOTO algorithm introduces innovative techniques like model-based value expansion, epistemic uncertainty corrections, and policy regularization.\n\n**Significance**: This work is good as it pushes the boundaries of reinforcement learning, particularly in the realm of robot tasks using high-dimensional observations. The MOTO algorithm's superior performance in comparison to existing methods demonstrates its potential impact in this field. However no real-robot experiment is conducted. The paper follows many existing methods from offline RL, model-based RL and imitation learning and well combines them under the offline pretraining and online fine-tuning setting.\n\n**Strengths**:\n1. **High Performance**: MOTO achieves better results than existing methods on several benchmark tasks, showcasing its effectiveness.\n2. **Efficient Data Use**: MOTO efficiently uses offline data for pre-training, making it more data-efficient than many existing methods.\n3. **Versatility**: The algorithm shows versatility by performing well on a variety of tasks.\n\n\n**Weaknesses**:\n1. **Limitations in Generalization**: The algorithm might struggle with generalization outside of the available data distribution. This limitation could affect its performance on tasks with unexpected or unseen scenarios.\n2. **Sparse Reward Tasks**: The approach may be difficult to adapt to tasks involving more complex or non-sparse rewards. \n3. **Transferability Concerns**: It is uncertain whether the scheme used to manage model-based epistemic uncertainty can transfer to more complex architectures, such as Transformers.\n4. **Lack of Real-World Testing**: The algorithm is tested on benchmarks and not in real-world settings, so its practical applicability and robustness are yet to be determined.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a model-based method for going from offline to online data in reinforcement learning. Model learning is performed on policy. In experimental evaluation in simulation tasks MetaWorld and Franka Panda Kitchen, the proposed approach is competitive w.r.t. comparison methods. The paper is well written.
Decision: Accept (Poster)