Model-Based Offline Meta-Reinforcement Learning with Regularization
Authors: Sen Lin
Jialin Wan
Tengyu Xu
Yingbin Liang
Junshan Zhang
Venue: ICLR 2022 Poster (2022)
Keywords: offline reinforcement learning
model-based reinforcement learning
behavior policy
Meta-reinforcement learning

Abstract: Existing offline reinforcement learning (RL) methods face a few major challenges, particularly the distributional shift between the learned policy and the behavior policy. Offline Meta-RL is emerging as a promising approach to address these challenges, aiming to learn an informative meta-policy from a collection of tasks. Nevertheless, as shown in our empirical studies, offline Meta-RL  could be outperformed  by offline single-task RL methods on tasks with good quality of datasets, indicating that a right balance has to be delicately calibrated  between "exploring" the out-of-distribution state-actions by following the meta-policy and "exploiting" the offline dataset by staying close to the behavior policy. Motivated by such empirical analysis, we propose model-based offline $\text{\bf Me}$ta-RL with $\text{\bf r}$egularized $\text{\bf P}$olicy $\text{\bf O}$ptimization (MerPO), which learns a meta-model for efficient task structure inference and an informative meta-policy for safe exploration of out-of-distribution state-actions. In particular, we devise a new meta-Regularized model-based Actor-Critic (RAC) method for within-task policy optimization, as a key building block  of MerPO, using both conservative policy evaluation and regularized policy improvement; and the intrinsic tradeoff therein is achieved via striking the right balance between two regularizers, one based on the behavior policy and the other on the meta-policy. We theoretically show that the learnt policy offers guaranteed improvement over both the behavior policy and the meta-policy, thus ensuring the performance improvement on new tasks via offline Meta-RL. Our experiments corroborate the superior performance of MerPO over existing offline Meta-RL methods.

Published as a conference paper at ICLR 2022
MODEL-BASED OFFLINE LEARNING WITH REGULARIZATION
META-REINFORCEMENNT
Sen Lin', Jialin Wan', Tengyu Xu², Yingbin Liang", Junshan Zhang
Schooo of ECEE, Arizona State University "Department of ECE, The Ohio State University
"Department of ECE, University of California, Davis
{slin70, jwan20) @ asu.edu, {xu.3260, liang. 889) @ osu.edu, jazh @ @ucdavis.edu
ABSTRACT
Existing offline reinforcement learning (RL) methods face few major challenges, particularly the distributional shift between the learned policy and the behavior policy. Offline Meta-RL. RL is emerging as a promising approach to address these challenges, aiming to learn an informative meta-policy from a collection of tasks. Nevertheless, as shown in our empirical studies, offline Meta-RI. could be outper- formed by offline single-task RI. methods tasks with good quality of datasets, indicating that a right balance has to be delicately calibrated between "exploring" the out-t--distribution state-actions by following the meta-policy and "exploiting" the offline dataset by staying close to the behavior policy. Motivated by such em- pirical analysis, we propose model-based offline Meta-RL. with regularized Policy Optimization (MerPO), which learns a meta model for efficient task structure in- ference and an informative meta-policy for safe exploration of -utto--distribution state-actions. In particular, we devise new meta-Regularizz model-based Actor- Critic (RAC) method for within-task policy optimization, as a key building block of MerPO, using both conservative policy evaluation and regularized policy im- provement; and the intrinsic tradeoff therein is achieved via striking the right bal- ance between two regularizers, one based on the behavior policy and the other the meta-policy We theoretically show that the learnt policy offers guaranteed improvement over both the behavior policy and the meta-policy, thus ensuring the performance improvement on new tasks via offline Meta-RL. Experiments corrob- orate the superior performance of MerPO over existing offline Meta-RL methods.
INTRODUCTION
Offline reinforcement learning (a.k.a,, batch RL) has recently attracted extensive attention by learn- ing from offline datasets previously collected via some behavior policy (Kumar et al. 2020). How- ever, the performance of existing offline RL methods could degrade significantly due to the following issues: 1) the possibly poor quality of offline datasets (Levine et al. 2020) and 2) the inability to generalize to different environments (Li et al. (020bb). To tackle these challenges, offline Meta-RL (Li al. 2020a Dorfman & Tamar 2020 Mitchell al. 2020 Li et al. 2020b) has emerged very recently by leveraging the knowledge similar offline RL. tasks ( Yuu et al.) [021a). The main aim of these studies is enable quick policy adaptation for new offline tasks, by learning meta-policy with robust task structure inference that captures the structural properties across training tasks.
Because tasks trained offline datasets, value overestimation (Fujimoto et al.) 2019) inevitably oc- curs in dynamic programming based offline Meta-RL, resulted from the distribution shift between the be- havior policy and the learnt task-specific policy. guarantee the learning performance on new offline tasks, a right balance has to be carefully calibrated be- tween "exploring" the out-of-distribution state-actions by following the meta-policy, and "exploiting" the of- fline dataset by staying close the behavior policy. However, such a unique fline Meta-RI approaches, which would likely limit their ability to handle diverse offline datasets
Waller (Bad deteset)
Walker (Cood detuset)
400
400
300
300
200
200
100
100
8.0
2.0
4.0
6.0
8.0
2.0
4.0
6.0
Gradient Stops (x30")
Gradient Staps (x110))
COMBO
FOCAL
Figure 1: FOCAL vs. COMBO.
tradeoff has not been considered existing of-

---

Published as a conference paper at ICLR 2022
particularly towards those with good behavior policies. To illustrate this issue more concretely, compare the performance between a state-of-thh-arr offline Meta-RL algorithm FOCAL (Li et al. 2020b) and offline single-task RL method COMBO (Yu et al., 2021b) in two new offline tasks. As illustraae in Figure 1 while FOCAL performs better than COMBO on the task with a bad-quality dataset (left plot in Figure D. it is outperformed by COMBO the task with a good-quality dataset (right plot in Figure It. Clearly, existing offline Meta-RL fails in several standard environments (see Figure and Figure 11 generalize universally well over datasets with varied quality, In order to fill such substantial gap, we seek to answer the following key question in offline Meta-RL:
How to design an efficient offline Meta-RL algorithm to strike the right balance between exploring with the meta-policy and exploiting the offtine dataset?
To this end, we propose MerPO, a modcl-based offline Meta-RL approach with regularized Policy Optimization, which learns a meta-model for efficient task structure inference and an informa- tive meta-ppliccy for safe exploration of disto--distiiuttion state-actions. Compared to existing ap- proaches, MerPO achieves: (I) safe policy improvement: performance improvement can be guar- anteed for offline tasks regardless of the quality of the dataset, by strike the right balance between exploring with the meta-policy and exploiting the offline dataset; and (2) better generulization ca- pability: through a conservative utilization of the learnt model to generate synthetic data, MerPO aligns well with a recently emerging trend in supervised meta-learning to improve the generaliza- tion ability by augmenting the tasks with "more data" (Rajendran et al. 2020 Yao et al. 2021). Our main contributions can be summarized follows:
(1) Learnt dynamics models not only serve as a natural remedy for task structure inference in of- fline Meta-RL, but also facilitate better exploration of out-of-distribbtion state-actions by generating synthetic rollouts. With this insight, we develop a model-based approach, where an offline meta- model is learnt to enable efficient task model learning for each offline task. More importantly, we propose a meta- regelaregue model-based actor-critic method (RAC) for within-tase -task policy opti- mization, where a novel regularized policy improvement module devised calibrate the unique tradeoff by using an interpolation between two regularizers, one based on the behavior policy and the other on the meta-policy. Intuitively, RAC generalizes COMBO to the multi-task setting, with introduction of novel regularized policy improvement module to strike a right balance between the impacts of the meta-policy and the behavior policy.
pnplloratio-exploiaaatio
(2) We theoretically show that under mild conditions, the learnt task-specific policy based on MerPO offers safe performance improvement over both the behavior policy and the meta-policy with high probability. Our results also provide guidance for the algorithm design in terms of how to ap- propriately select the weights in the interpolation, such that the performance improvement can be guaranteed for new offline RL tasks.
(3) We conduct extensive experiments to evaluate the performance of MerPO. More specifically, the experiments clearly show the safe policy improvement offered in MerPO, corroborating our theo- retical results. Further, the superior performance of MerPO over existing offline Meta-RL methods suggests that model-based approaches be more beneficial in offline Meta-RI..
2 RELATED WORK
Offline single-task RL. Many existing model-free offline RL methods regularize the learnt policy to be close to the behavior policy by, e.g., distributional matching (Fujimoto al. 2019), support matching (Kumar et al. 2019), importance sampling (Nachum et al. 20u et al. 2020), learning lower bounds of true Q valuuess (Kumar et al.] 2020). Along a different avenue, model-based algo- rithms learn policies by leveraging a dynamics model obtained with the offline dataset. (Matsushima al. 2020) directly constrains the learnt policy to the behavior policy as in model-free algorithms. To penalize the policy for visiting states where the learnt model is likely to be incorrect, MOPO (Yu et al. 2020) and MoREL (Kidambi et al. 2020) modify the learnt dynamics such that the value estimates conservative when the model uncertainty is above a threshold. To remove the need of uncertainty quantification, COMBO YYu et al. 2021b) is proposed by combining model-based policy optimization (Janner et al. 2019) and conservative policy evaluation (Kumar et al. 2020).
Offline Meta-RL. A few very recent studies have explored the offline Meta-RL. Particularly, et al. 2020a) considers special scenario where the task identity is spuriously inferred due to biased
2

---

Published as a conference paper at ICLR 2022
datasets, and applies the triplet loss to robustify the task inference with reward relabelling. (Dorfman & Tamaar 2020) extends an online Meta-RL method VariBAD Zintgraf et al. 2019) to the offline setup, and assumes known reward functions and shared dynamics across tasks. Based on MAML (Finn et al. 2017), (Mitchell et al., 2020) proposes an offline Meta-RI. algorithm with advantage weighting loss, and learns initializations for both the value function and the policy, where they consider the offline dataset in the format of full trajectories in order to evaluate the advantage. Based on the off-policy Meta-RL method PEARL (Rakelly al. 2019), (Li et al. 2020b) combines the idea of deterministic context encoder and behavior regularization, under the assumption of deterministic MDP. Different from the above works, we study a more general offline Meta-RL problem. More importantly MerPO strikes right balance between exploring with the meta-policy and exploiting the offiin dataset, which guarantees safe performance improvement for new offline tasks.
PRELIMINARIES
Consider a Markov decision process (MDP) M = (S,A,T, po.
7) with state space S, action
space A, the environment dynamics T(s s, a), reward function r(s, a). initial state distribution
Mo. and E (0, is the discount factor. Without loss of generality, we assume that (r(s.a) <
Rmaz- Given a policy H. let (s) == (1 -1-10 7) Ei=o re PM = s/T) denote the discounted marginal state distribution, where PM((s s/TT) denotes the probability of being in state s at time 1 by rolling out л in M. Accordingly, let d = dT)(s) denote the discounted marginal state-action distribution, and ((M, .T == I
T- (saa))) (z (r(s.a) denote the expected discounted return. The goal of RL is to find the optimal policy that maximizes J[M, .T. In offline RL, no interactions with the environment are allowed, and we only have access to a fixed dataset D ((8, a, r, } collected by some unknown behavior policy my.. Let (s) be the discounted marginal state distribution of TT. The dataset D is indeed sampled from d a) dMM (s) m (als). Denote M the empirical MDP induced by D and d(ss, a) as sample- based version of d'M (s, a).
In offline Meta-RL, RL, consider a distribution of RL tasks p(M) as in standard Meta-RL (Finn et al, 2017 Rakelly et al. 2019), where each task M, is an MDP, i..., M, AS, Tr, T's Mo.n: 7), with task-shared state and action spaces, and unknown task-specific dynamics and reward function.For each task Mn. no interactions with the environment offline dataset Dn, collected by some unknown behavior policy Tp,.- The main objective is to learn a meta-policy based on a set of offline training tasks {M,n)
allowed and we only have access to an
Conservative Offline Based Policy Optimization (COMBO), Recent model-based offline RL algorithms, e.g., COMBO (Yu et al.) (0211)), have demonstrated promising performance on single offline RL task by combining badel-based policy optimization (Janner et al. 2019) and conservative policy evaluation (CQL (Kumar et al., 20200). Simply put, COMBO first trains a [s. a) parameterized by 8, via supervised learning on the offline dataset D. The learnt MDP is constructed as M == AS A, ri poo 7). Then, the policy is learnt using D and model-generated rollouts. Specifically, define the action-value function (Q-function) (=(ss ==
dynamics model (s(5'|s,
E D > ('(s,, {s) = s, ag = a). and the empirical Bellman operator as: B (({(s, a) = r(s, a)+ (a.,"""")> (anii//(sss d'). for a' ~ (--1s"). To penalize the Q functions in duito--distribution state- action tuples, COMBO employs conservative policy evaluation based on CQL:
@i+1 arg min 3(E.,a (Q(s, a)] - al a))) + 2 _ [(Q(s. a) 8 (s, a)) (1)
((s.a)
where p(s,a) == d (a)s) is the discounted marginal distribution when rolling out æ in M,
and (s. = fd a) + (1 f)p(s, a) for f E 10. The Bellman backup 8- over d, can be interpreted an Jinterppolation of the backup operators under the empirical MDP (denoted by B and the learnt MDP (denoted by T. Given the Q estimmation Q", the policy can be learnt by:
arg max (t (².(s. (s.
4 MERPO: MODEL-BASED OFFLINE META-RL WITH REGULARIZED POLICY OPTIMIZATION
Learnt dynamics models not only serves as a natural remedy for task structure inference in offline Meta-RI., but also facilitates better exploration of out-of-distribution state-actions by generating
3

---

Published as a conference paper at ICLR 2022
policy optimization
offline meta-model estimation
Task 1
task model estimation
policy optimization
Task
meta- model
policy evaluation
policy
Task N
task model estimation
policy optimization
policy
improvement
Tisk
offline meta-policy update
offline mela-training tasks
Figure 2: Model-based offline Meta-RL. with learning of offline meta-modes and offline meta-policy.
synthetic rollouts Yu et al. 2021b Thus motivated, we propose a general framework of model- based offine Meta-RI., depicted in Figure More specifically, the offline meta-model is first learnt by using supervised meta-learning, based on which the task-specific model can be quickly adapted. Then, the main attention of this study is devoted the learning of informative meta- policy via bi-level optimization, where 1) model-based policy optimization approach is leveraged in the inner loop for each task to learn a task-specific policy; and 2) the meta-policy is then updated in the outer loop based on the learnt task-specific policies.
4.1 OFFLINE MODEL LEARNING
Learning meta-model based on the set of offline dataset (Dr) can carried out via supervised meta-learning. Many gradient-basse based meta-learning techniques can be applied here, e.g., MAMI. (Finn et al. 2017) and Reptile (Nichol et al. 2018). In what follows, we outline the basic idea to leverage the higher-order informatron of the meta-objective function. Specifically, we consider proximal meta-learning approach, following the same line as in (Zhou al.) 2019):
min Lmndea((o)) = EM,,, min dm where the learnt dynamics for each task M, is parameterized by On and the meta-model is parame- terized by @mm Solving Eq. leads an offline meta-model.
[(c.a..")> [log Te. (s'Is, a)] vl 3m.
(3)
Given the learnt meta-model Te:.. the dynamics model for an individual offline task j can be found by solving the following problem via gradient descent with initialization To: using Dj. i.e.
min E(s.a,e))-D, [log Tey (s' [s. a)] + v/|00 - dm.1..
(4)
8;
Compared to learning the dynamics model from scratch, adapting from Té- can quickly generate a dynamics model for task identity inference by leveraging the knowledge from similar tasks, and hence improve the sample efficiency (Finn et al. 2017 Zhou et al. 2019).
4.2 OFFLINE META-POLICY LEARNING
In this section, we turn attention to tackle one main challenge in this study: How to learn an infor- mative offline meta-policy order to achieve the optimal tradeoff between "exploring" the out-of- distribution state-actions by following the meta-policy and "cxploiting" the offline dataset by staying close to the behavior policy? Clearly, highly desirable for the meta-policy to safely 'explore' duitr--disttibution state-action pairs, and for each task utilize the meta-policy to mitigate the issue of value overestimation.
4.2.1 How Do EXISTING PROXIMAL META-RL RL APPROACHES PERFORM?
Proximal Meta-RL. approaches have demonstrated remarkable performance in the online setting (e.g.. [Wang et al. (2220), by explicitly regularizing the task-specific policy close to the meta-policy. We first consider the approach that applies the online Proximal Meta-RL method directly to devise offline RL, which would lead to:
[Q%(s..) a)] AD(TT, me)
max BM max
R.
(5)
where म. is the offline meta-policy, n. is the task-specific policy, Pn. is the state marginal of pm (s, a) for task n and D(.,)) is some distance measure between two probability distributions. To alleviate value overestimation, conservative policy evaluation can be applied to learn Q% by using Eq. (1)
4

---

Published as a conference paper at ICLR 2022
Intuitively, Eq. [ corresponds to generalizing COMBO to the multi-task setting, where meta policy Te is learned to regularize the within-task policy optimization.
To get sense of how the meta- meta-policy learnt using Eq. 5 performs, we evaluate its performance in an offline variant of standard Meta-RL. RL benchmark Balkeer-2D-Paramee with good-quality datasets, and evalu- ate the testing performance of the task-specific policy after fine-tuning based on the learnt meta-policy, with respect to the meta-training steps. As can be seen in Figure 3 the proximal Meta-RL algorithm Eq. (5) performs surprisingly poorly and fails to learn an informative meta-policy, despite conservative policy evaluation being applied in within-task policy optimization to deal with the value overestimation. In particular, the testing performance degrades along with the meta- training process, implying that the quality of the learnt meta-policy is in fact decreasing.
3 gee2
Welker-2O-Paramms
Awerage netum
0.0
10
2.0
3.0 4.0 5.0
Gradient Steps (x10")
Proocrmet Neka-iti Eqge5)
Figure 3: Performance of proximal Meta-RL Eq. 5
Why does the proximal Meta-RL method in Eq. (5) perform poorly in offline Meta-RL, even with conservative policy evaluation? To answer this, it is worth to take a closer look at the within-task policy optimization in Eq. (5) which is given as follows:
(6) Clearly, the performance of Eq. (6) depends heavily on the quality of the meta-policy Te. A poor meta-policy may have negative impact on the performance and result in task-specific policy n, that even outperformed by the behaviour policy Tg Without online exploration, the quality of nn could not be improved, which in turn leads to worse meta-policy T. through Eq. (5). The iterative meta-training process would eventually result in the performance degradation in Figure 3
Tha arg max Ea-pp. (-1a) [Q%
AD(T,, Te).
In nutshell, simply following the meta-policy may lead to worse performance of offline tasks when Ts is a better policy than Te. Since it is infeasible to guarantee the superiority of the meta-policy priori, it is necessary to balance the tradeoff between exploring with the meta-policy and exploiting the offline dataset, in order to guarantee the performance improvement new offline tasks.
4.2.2 SAFE POLICY IMPROVEMENT WITH META- REGULARIZATION
To tackle the above challenge, we next devise a novel regularized policy improvement for within- task policy optimization of task n, through a weighted interpolation of two different regularizers based on the behavior policy TH, and the meta-policy Te, given as follows:
f arg max Bes-p..aat,t
( (s, a)] XaD(TT, Tp...) A(1 a) Te),
for some E [0, 1]. Here, a controls the trade-off between staying close to the behavior policy and following the meta-policy to "explore" out-of-distribution state-actions. Intuitively, as a is closer to O, the policy improvement is less conservative and tends to improve the task-specific policy on towards the actions in n. that have highest estimated Q-values. Compared to Eq. (6). the exploration penalty induced by D(T,, Tp... serves a safeguard and stops n, following ne over-optimistically.
Safe Policy Improvement Guarantee. Based on con- servative policy evaluation Eq. (I) and regularized pol- icy improvement Eq. 7 we have the meta-regularized mosel-based actor-critic method (RAC), as outlined in Algorithm
Algorithm 1 RAC
1: Train dynamics model To, using D.:; for k= 1,2,... do
2:
[ Note that different distribution distance measures can be used in Eq. 7 In this work, we theo- retically show that the policy n. (a]s) learnt by RAC is safe improvement over both the behavior policy T3,n and the meta-policy Te on the underlying MDP M,, when using the maximum total-variation distance for D(1,,, 12), i.e., D(TI, 72) == max, (m
3: Perform model rollouts starting from states in D, and add into moodel Policy evaluation by recursively solv-
4:
ing Eq. (T) using D, U mondet,n
5:
Improve policy by solving Eq.
6:
end for
For convenience, define vn f) E. ()(s. - ds dr. (s, a)), and let 5 E 1/2). We have the following important result on the safe policy improvement achieved by (a)s).
Theorem 1. (a) Let = (pr "f0"("")_~000000""" (00m.() If "
v tp f) > 0 and a e (max{{ + e, 0}. 4) then J(M, n TT) 2 max(J(M Tt) + E1. (MM Tz + E2) holds with probability least 25, where both E and &a are positive for large enough $3 and X;
2A(1-3)((an
5

---

Published as a conference paper at ICLR 2022
(b) More generally, we have that J TT) 2 max{ Te) + J =B,n) + F2) holds with probability at least 1 - 26, when a E (0, (021 where & is positive for large enough X.
Remark 1. The expressions of 5 and &2 are involved and can be found in Eq. 14 and Eq. 15 in the appendix. In part (a) of [ both & and &2 are positive for large enough [3 and X, pointing to guaranteed improvements over me and zg Due to the fact that the dynamics Typ, learnt via supervised learning is close the true dynamies TM,, on the states visited by the behavior policy MM
T8,n. df d.A..n (s, is close to d d (s. a) and pT*." is close to d, (s, a), indicating that the condition
'M..
"n(p" : f))-u (p= J) > 0 expected to hold in practical scenarios (Yu et al. 202Tb). For more general cases, slightly weaker result can be obtained in part (b) of Theorem where & is positive for large enough and &2 can be negative.
Intuitively, the selection of a balances the impact of Tæ and Te, while delicately leaning toward the meta-policy Te because Ts has played an important role in policy evaluation to find a lower bound of Q-value. As result, Eq. (7) maximizes the true Q-value while implic- itly regularized by a weighted combination, instead of o-interpolattin, between D(=;: R,,)) and D(T,, Re), where the weights carefully balanced through In particular, in the tabular setting, the conservative policy evaluation in Eq. (I) corresponds penalizing the Q estimation (Yu al., 2021b):
Remark 2.
@ B - Blp(s,a) a) - as]
B
(8)
(1,n(
Clearly, c increases with the value of the penalty term in Eq. (B. As a result, when the policy evaluation Eq. [I] is overly conservative, the lower bound of will be close to O, and hence the regularizer based on the meta-policy Te can play a bigger role so as to encourage the "exploration" of out-of-distributioo state-actions following the guidance of Te. On the other hand, when the policy evaluation Eq. (1) is less conservative, the lower bound of a will be close to 5. and the regularizer based on JT will have more impact, leaning towards "exploiting" the offline dataset. In fact, the introduction of 1) behavior policy-basee regularizer and 2) the interpolation for modeling the interaction between the behavior policy and the meta meta-policy, is the key to prove Theorem [|
Practical Implementation. In practice, we can use the KL divergence to replace the total variation distance between policies, based on Pinsker's Inequality: || T22 S 222x kL (Ta [ Moreover, since the behavior policy TT. is typically unknown, we can use the reverse KL- KL-divergence between Tn and Jzn to circumvent the estimation of 7 following the same line in (Fakoor et al. 2021 (f
Ea [log Tp... (als) Ea-Ts [log nn (als)
-Bawwg.. p. log n.. (a|s) [s] ~ - -,, [log n..
Then, the task task-spec policy can be learnt by solving the following problem:
max F. Ex-pa ( a)] +
[log ( (a]=)) - X(1 - a.) DxL Dx(
(9)
4.2.3 OFFLINE META-POLICY UPDATE
Built on RAC, the offline meta-policy ne is updated by taking the following two steps, in an iterative manner: 1) (inner loop) given the meta-policy n., RAC is run for each training task to obtain the task-specific policy nni 2) (outer loop) based on } Te is updated by solving:
max TE,, { empey [QT(o.a)] (o, [log t. (als)] A(1 - Da) (TnllTe))
(10)
where both Pn and Q are from the last iteration of the inner loop for each training task. By using RAC in the inner loop for within-task policy optimization, the learnt task-specific policy on and the meta-policy ne work in concert to regularize the policy search for each other, and improve akin to 'positive feedback'. Here the regularizer based on the behavior policy serves an important initial force to boost the policy optimization against the ground: RAC in the inner loop aims to improve the task-specific policy over the behavior policy at the outset and the improved task-specific policy consequently regularizes the meta- peea--policy search as in Eq. (T0), leading to a better meta-policy eventually. Noted that a meta-Q network is learnt using first-order meta- meta-llerring to initialize task- specific Q networks. It is worth noting that different tasks can have different values of a to capture the heterogeneity of dataset qualities across tasks.

---

Published as a conference paper at ICLR 2022
1e3 PoitChocta
Jnc Hopper Modium Replay
1n3. HollCheetah Hedium
1n3 Hoppor Medium Repley
Avenge Retum
Auaaago Rotum
M
Arergg
100
400
400
200 Epicode
300
Ep sode
Episade
Egisode
Figure 4: Performance comparison among COMBO, COMBO-3 and RAC, with good-quality meta- policy (two figures on the left) and poor-quality meta-policy (two figures the right).
In nutshell, the proposed model-based offline Me with regularized Policy Optimization (MerPO) is built on two key steps: 1) learning the offline meta-model via Eq. (3) and 2) learning the
offline meta-policy via Eq. 10 The details are presented in Algorithm 2 in the appendix.
4.3 MERPO-BASER POLICY OPTIMIZATION FOR NEW OFFLINE RL TASK
Let To: and T. be the offline meta-model and the offline meta-policy learnt by MerPO. For new offline RL task, the task model can be quickly adapted based Eq. (4) and the task-specific policy can be obtained based on π. using the within-tan task policy optimization module RAC. Appealing to Theorem we have the following result on MerPO-based policy learning on a new task.
Proposition 1. Consider new offline RL task with the true MDP M. Suppose To is the -based task-specific policy, learnt by running RAC over the meta-policy n.. If Blvi0" "M -f=
2X(1 2X(1-3)D(Te 2 0 and a E (max( - e, 03, h)) then To achieves the safe performance improvement over both τ² and np, i.e., (MM, To) > max(J(M, (M, π&)} holds with prob- ability at least 26, for large enough and X.
Proposition D Dindicates that MerPO-based policy optimization for learning task task-specific policy guar- antees policy with higher rewards than both the behavior policy and the meta-policy. This is particularly useful in the following two scenarios: 1) the offline dataset is collected by some poor behavior policy, but the meta-policy is good policy; and 2) the meta-policy is inferior good behavior policy.
EXPERIMENTS
In what follows, we first evaluate the performance of RAC for within-task policy optimization on offline RL task validate the safe policy improvement, and then examine how MerPO performs when compared to state-of-art offline Meta-RI. algorithms. Due to the space limit, we relegate additional experiments to the appendix.
5.1 PERFORMANCE EVALUATION OF RAC
Setup. We evaluate RAC on several continuous control tasks in the D4RL benchmark (uu et al. 2020) from the Open AI Gym (Brockman et al. (016),, and compare its performance to COMBO (where no meta meta-policy is leveraged) and 2) COMBO with policy improvement Eq. (6) (namely, COMBO-3), under different qualities of offline datasets and different qualities of meta-policy (good and poor). For illustrative purpose, we use random policy as poor-quality meta-policy, and choose the learnt policy after 200 episodes as better-quality meta-policy. We evaluate the average return over 4 random seeds after each episode with 1000 gradient steps.
Results. As shown in Figure 4 RAC can achieve comparable performance with COMBO-3 given good-quality meta-policy, and both clearly outperform COMBO. Besides, the training procedure also more stable and converges more quickly as expected when regularized with the meta- meta-policy. When regularized by poor-quality meta-polian that is significantly worse than the behavior policy in all environments, the performance of COMBO-3 degrades dramatically. However, RAC outper- forms COMBO even when the meta-policy is random policy. In nutshell, RAC consistently

---

Published as a conference paper at ICLR 2022
achieves the best performance in various setups and demonstrates compelling robustness against the
quality of the meta-policy, for suitable parameter selections (a = 0.4 in Figure
Impact of As shown in Theorem I the selection of a is important to guarantee the safe policy improvement
HollChnetth-Boondom
HollCheeeah-Meddum
Average Ratum
Avrrage Return
property of RAC. Therefore, we next examine the impact of on the per- formance of RAC under different qual- ities of datasets and meta-policy. More specifically, we consider four choices of a: a O, 0.4, 0.7, Here, O corresponds to COMBO-3, i.e., reg- ularized by the meta-poliey only, and the policy improvement step is regular- ized by the behavior policy only when
2 3
2
0
300
200
300
Episode
Episode
dote-b.c
elcte-1.0
dote-bb
Figure 5: Impact of a on the performance of RAC under different qualities of offline datasets.
1. Figure | shows the average return of RAC over different qualities of meta-policies under different qualities of the offline datasets. is clear that RAC achieves the best performance when
0.4 among the four selections of ot, corroborating the result in Theorem [ In general, the performance of RAC is stable for a E [0.3, 0.5] in our experiments.
5.2
PERFORMANCE EVALUATION OF MERPO
Setup. To evaluate the performance of MerPO, we follow the setups in the literature (Rakelly et al., 2019 Li et al. (2020b) and consider continuous control meta-environments of robotic locomotion. More specifically, tasks has different transition dynamies in Walker-2D 2aare-DDPaaaasss and Point-Robot- Wind, and different reward functions in Half-Cheetth--Fwd--
and Ant-Fwd-Back. We collect the offline dataset for each task by following the same line in (Li et al. 20206). We consider the following baselines: (1) FOCAL (Li et al. 2020b), a model-free offline Meta-RL. approach based on deterministic context encoder that achieves the state-of-the-ar performance: (2) MBML (Li et al. 2020a), an offline multi-task RL approach with metric learning; (3) Batch PEARL, which modifies PEARL (Rakelly et al., 2019) to train and test from offline datasets without exploration; (4) Contextual BCQ (CBCQ), which is a task- augmented variant of the offline RL algorithm BCQ (Fujimoto al. 2019) by integrating a task latent variable into the state information. We train on set of offline RI. tasks, and evaluate the performance of the learnt meta-policy during the training process on a set of unseen testing offline RL tasks.
We consider two implementations of MerPO based on the selection of a. 1) MerPO: a is fixed as 0.4 for all tasks; 2) MerPO-Adp: at each iteration k, given the task-policy Th for task n and the meta-policy oh at iteration k, we update a using one-step gradient descent to minimize the following problem.
Fixed Adaptive
min (1 - (D( TB..) - D(= +))), s.t. an E 0..1,
(1I)
The idea is to adapt a in order to balance between D(= n) and D(= x²), because Theorem Timplies that the safe policy improvement can be achieved when the impacts of the meta-policy and the behavior policy are well balanced. Specifically, at iteration & for each task n, o is increased when the toskcy T is closer to the meta-policy = and is decreased when = is closer to the behavior policy. Note that a is constrained in the range [0.1, 0.5] suggested by Theorem [
Results. As illustrated in Figure 6 MerPO-Adp yields the best performance, and both MerPO- Adp and MerPO achieve better or comparable performance in contrast to existing offline Meta-RL approaches. Since the meta-policy changes during the learning process and the qualities of the behavior policies vary across different tasks, MerPO-Adp adapts across different iterations and tasks so as to achieve a 'local' balance between the impacts of the meta-policy and the behavior policy. As expected, MerPO-Adp testing performance for the baseline algorithms selected over different qualities of offline datasets.
perform better than MerPO with fixed Here the best
Ablation Study. We next provide ablation studies by answering the following questions.
(1) Is RAC important for within-task policy optimization? To answer this question, we compare MerPO with the approach Eq. (5) where the within-task policy optimization is only regularized by
8

---

Published as a conference paper at ICLR 2022
Te2
We Bear Pa-Pare
Jel Hall ChattaChstaaaaat Bed
162
AntiPord Bard
Point Releot Wind
Average Return
Aueer Raturn
Averrge Return
0.8
Average
-6
9.4
0.0
5.0
0.0
3.0
5.0
0.0
5.0
0.0
1.0
Gradiont
Gradient
Gradiant
Gradient
(x107)
CSCO
C3CO
CSCO
CXCO
Figure 6: Performance comparison in terms of the average return in different environments. Clearly, MerPO Adp and MerPO achieve better or comparable performance than the baselines.
Wa kor
Waiker 20 Paramn
Anarage Betumn
Average Retum
Rive 2000 Retum
0.0 10 2.0
3.0
5.0
0.0
1.0
20
3.
4.0
5.0
0.0
4.0 Gredient Steps (x10%)
1.0
2.0
3.0
5.0
2.0
4,0
6.0
Gradient Steps 1x10") with zana megil
Gredient Steps 01107)
Gradient Steps bd10
-Wth mete ZRMAn With mate TOOk.
migal.
mode Without
With mart
only
(a) Impact of RAC mod- (h) Impact of model uti- (c) Performance under dif- (d) Testing performance ferent data qualities.
ule.
lization.
for expert dataset.
Figure 7: Ablation study of MerPO in Walker-2D-Parass.
the meta-policy. As shown in Figure 7(a) with the regularization based on the behavior policy in
RAC, MerPO performs significantly better than Eq. (5). implying that the safe policy improvement property of RAC enables MerPO to continuously improve the meta-policy.
(2) Is learning the dynamics model important? Without the utilization of models, the within-task policy optimization degenerates to CQL (Kumar et al. 2020) and the Meta-RL algorithm becomes a model-free approach. ((b))re(((sshhaa) shows the performance comparison between the cases whether the dynamics model is utilized. It can be seen that the performance without model utilization is much worse than that of MerPO. This indeed makes sense because the task identity inference (Dorfman & Tamar 2020 Li et al.) 2020a b) is a critical problem in Meta-RL. Such a result also aligns well with a recently emerging trend in supervised meta-learnng to improve the generalization ability by augmenting the tasks with "more data" (Rajendran et al. 2020 Yao et al. 2021).
(3) How does MerPO perform in unseen offline tasks under different data qualities? We eval- uate the average return in unseen offline tasks with different data qualities, and compare the per- formance between (1) MerPO with a 0.4 ("With meta") and (2) Run a variant of COMBO with behavior-regularizze policy improvement, i.e., 1 ("With beha only"). For fair comparison, we initialize the policy network with the meta-policy both cases. As shown in Figure age performance of "With meta" over different data qualities is much better than that of "With beha only". More importantly, for a new task with expert data, MerPO W'iith meta") "eeta") clearly outperforms COMBO illustrated in Figure 7 whereas the performance of FOCAL is worse than COMBO.
[<c the aver-
CONCLUSION
In this work, we study offline Meta-RL aiming to strike right balance between "exploring" the oft-of-distributton state-actions by following the meta-policy and "exploiting" the offline dataset by staying close to the behavior policy. To this end, we propose model-based offline Meta-RL approach, namely MerPO, which learns a meta-model to enable efficient task model learning and a meta-policy to facilitate safe exploration of out-of-distributiin state-actions. Particularly, devise RAC, meta-regularized model-based actor-critic method for within- task policy optimization, by using weighted interpolation between two regularizers. one based on the behavior policy and the other on the meta-policy. We theoretically show that the learnt task-policy via MerPO offers safe policy improvement over both the behavior policy and the meta-policy. Compared to existing offline Meta-RL. methods, MerPO demonstrates superior performance on several benchmarks, which suggests a more prominent role of model-based approaches in offline Meta-RL.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work is supported in part by NSF Grants CNS-2003081, CNS-2203239, CPS-1739344, and CCSS-2121222.
REPRODUCIBILITY STATEMENT
For the theoretical results presented in the main text, state the full set of assumptions of all theoretical results in Appendix B and include the complete proofs of all theoretical results in Ap- pendix C For the experimental results presented in the main text, include the code in the supple- mental material, and specify all the training details in Appendix A For the datasets used in the main text, we also give clear explanation in Appendix
REFERENCES
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.0/54 2016.
Ron Dorfman and Aviv Tamar. arXiv:2008. 02598, 2020.
Offline
meta reinforcement learning.
arXiv preprint
Rasool Fakoor, Jonas Mueller, Pratik Chaudhari, and Alexander J Smola. Continuous doubly con- strained batch reinforcement learning. arXiv preprint arXiv:2102.0911 102.09925, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072/9, 2020.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2052-2063 PMLR, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. arXiv preprint arXiv: vexiv:/906.0 190.:02903,, 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint arXiv:2005.05951, 2020.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv: 1906600906.999 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006. 044779900.04779, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Jiachen Li, Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Henrik Christensen, and Hao Su.
Multi-task batch reinforcement learning with metric learning. Advances in Neural Information Processing Systems, 33, 2020a.
Lanqing Li. Rui Yang, and Dijun Luo. Efficient fully-offline meta-reiiforcemeent learning via dis- tance metric learning and behavior regularization. arXiv preprint 1/X1v:2010.01//2, 2020b.
10

---

Published as a conference paper at ICLR 2022
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-poliog policy gradient with stationary distribution correction. In Uncertainty Artificial Intelligence, pp. 1180-11900 PMLR, 2020.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. arXiv preprint arXiv:2006.036477 2020.
Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. arXiv preprint arXiv:2008. 2020.
Ofir Nachum, Bo Dai, Ila Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074 2019.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803. 02999, 2018.
Janarthanan Rajendran, Alex Irpan, and Eric Jang. Meta-learning requires meta-augmentation. arXiv preprint arXiv:2007.005593 2020.
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off off-po reta-reiinorremment learning via probabilistic context variables. arXiv preprint 1rXiv:19030082:: 082544 2019.
Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. On the global optimality of model- agnostic meta-learning In International Conference on Machine Learning, pp. 9837-9846. PMLR, 2020.
Huaxiu Yao, Long- Huang, Linjun Zhang, Ying Wei, Li Tian, James Zou, Junzhou Huang, et al. Improving generalization in meta-learning via task augmentation. In International Conference Machine Learning, pp. 11887-11897, PMLR, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea
Finn, and Tengyu Ma.
Mopo: Model-based offline policy optimization. arXiv preprint
1rXiv:2005.13239 2020.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. arXiv preprint arXiv:2109.08/28, 2021a.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint arXiv:2102.08363, 2021b.
Pan Zhou, Xiaotong Yuan, Huan Xu, Shuicheng Yan, and Jiashi Feng. Efficient meta learning via minibatch proximal update. Advances in Neural Information Processing Systems, 32:1534-1544, 2019.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta- learning. arXiv preprint 1rXiv:1910.088448.00 2019.

---

Published as a conference paper at ICLR 2022
Table Hyperparameters for RAC.
Hyperparameters
Halfcheetah
Hopper
Walker2d
Discount factor
0.99
0.99
0,99
Sample batch size
256
256
256
Real data ratio
0.5
0.5
0.5
Model rollout length
5
5
1
Critic Ir
3e-4
3e-4
le-4
Actor Ir
1e-4
le-4
le-5
Model
le-3
le-3
le-3
Optimizer
Adam
Adam
Adam
1
10
Max entropy
True
True
True
1
1
A EXPERIMENTAL DETAILS
A.1 META ENVIRONMENT DESCRIPTION
Salker-2D-Paramtee Train agent move forward. Different tasks correspond different
randomized dynamcis parameters.
Half- Salf-Cheeta-Fyee Fwd-Backk ward function depends on the moving direction. All tasks have the same dynamics model but different reward functions.
Train Cheetah robot to move forward or backward, and the
Ant-Fwd- aat-Fwd-Bacck Back: Train an Ant robot to move forward or backward, and the reward function depends on the moving direction. All tasks have the same dynamics model but different reward functions.
Poing- Doing-Robot-o Wind: Point-Robot- Wind is variant of Sparse- Point-Robot (Li et al. 2020b),
a 2D navigation problem introduced (Rakelly et al. 2019), where each task is to guide a point robot to navigate to specific goal location on the edge of semi-circle from the origin. In Point-Robot Wind, each task is affected by a distinct "wind" uniformly sampled from |-0.55, 0.05]², and hence differs in the transition dynamics.
A.2 IMPLEMENTATION DETAILS AND MORE EXPERIMENTS
A.2.1 EVALUATION OF RAC
Model learning. Following the same line as in (Janner et al.) 2019 Yu et al., 2020 2021b), the dynamics model for each task is represented as a probabilistic neural network that takes the current state-action input and outputs Gaussian distribution over the next state and reward:
Ta a) / -Nuno ae): (s az)).
An ensemble of 7 models is trained independently using maximum likelihood estimation, and the best models are picked based on the validation prediction error using a held-our set of the offline dataset. Each model is represented by a 4-layer feedforward neural network with 256 hidden units. And one model will be randomly selected from the best models for model rollout.
Policy optimization. We represent both Q-network and policy network 4-layer feedforward neural network with 256 hidden units, and use clipped double Q-learning (Fujimoto et al. 2018)) for Q backup update. A max entropy term is also included to the value function for computing the target Q value as SAC (Haarnoja et al. (018). The hyperparameters used for evaluating the performance of RAC are described in Table [
Additional experiments. We also evaluate the performance of RAC in Walker2d under different qualities of the meta-policy. As shown in Figure 8(a) and 8(b) RAC achieves the best performance in both scenarios, compared to COMBO and COMBO-3 3. Particularly, the performance COMBO- 3 in Figure ((a) degrades in the later stage of training because the meta-policy is not superior over the behavior policy in this case. In stark contrast, the performance of RAC is consistently better, as provides safe policy improvement guarantee over both the behavior policy and the meta-policy.
12

---

Published as a conference paper at ICLR 2022
1e3 Walker 24 Medium
1m3
Walker2d Modium
led. HallChectanEgee Expect
Jel
Export
1.2
Aieage Retum
Ayarage Raturn
Eowsage Betum
0.0
Avenge Rotum
1.0
0.8
20.5
200
100
200
400
100
430
100
200 Epicade
300
400
Epicade
Egisode
Episode
(a)
Performance (d) Average return over Walker2d with a good Walker2d with a random HalfCheetah with expert different qualities of offline dataset.
Performance in (b)
Performance in (c)
meta- mola-y.licy.
meta- meta-policy.
meta-policies under ex- pert dataset for different choices
Figure 8: Performance evaluation of RAC.
Beside, we also compare the performance of these three algorithms under an expert behavior policy in Figure 8(c) where a meta-policy usually interferes the policy optimization and drags the learnt policy away from the expert policy. As expected, RAC can still achieve comparable performance with COMBO, as a result of safe policy improvement over the behavior policy for suitable parameter selections.
We examine the impact of a on the performance of RAC under different qualities of the meta-policy for HalfCheetah with expert data. In this case, the meta-policy is worse policy compared to the behavior policy. As shown in Figure ((d) the performance a = 0.4 comparable to the case of I where the policy improvement step is only regularized based on the behavior policy, and clearly better than the other two cases.
A.2.2 EVALUATION OF MERPO
Data collection. We collect the offline dataset for each task by training a stochastic policy network using SAC (Haarnoja et al. 2018) for that task and rolling out the policies saved each checkpoint to collect trajectories. Different checkpoints correspond to different qualities of the offline datasets. When training with MerPO, we break the trajectories into independent tuples {5a, a, rs, s and store in replay buffer. Therefore, the offline dataset for each task does not contain full trajectories over entire episodes, but merely individual transitions collected by some behavior policy.
Setup. For each testing task, obtain the task-specific policy through quick adaptation using the within-task policy optimization method RAC, based on its own offfine dataset and the learnt meta- policy, and evaluate the average return of the adapted policy over 4 random seeds. As shown earlier, we take a = 0.4 for all experiments about MerPO. In MerPO-Adp, we initialize a with 0.4 and update with a learning rate of 1e 4.
Meta-model learning. Similar as in section A.2. for each task we quickly adapt from the meta- model obtain an ensemble of 7 models and pick the best models based the validation error. The neural network used for representing the dynamics model is same with that in section 1.2.
Meta-policy learning. As in RAC, we represent the task Q-network, the task policy network and the meta-policy network as a 5-layer feedforward neural network with 300 hidden units, and use clipped double Q-learning (Fujimoto we also use dual gradient descent to automatically tune both the parameter / for conservative policy evaluation and the parameter for regularized policy improvement:
al. 2018) for within task Q backup update. For each task,
Tune 8. Before optimizing the Q-network in policy evaluation, first optimize 3 by solving the following problem:
min max ((E...>p|() a)) B..aD (Q -7) B Be, ((Q(s, 8 (s, a))?).
Intuitively, the value of /3 will be increased to penalty the Q values for df-ttrribuutiinn state- actions if the difference E, E (1(s, -E, (Q,s, )] is larger than some thresh- old value т.
13

---

Published as a conference paper at ICLR 2022
Table 2: Hyperparameters for MerPO.
perpearraeeersss
Saikeer--D-Paa PD-Parmss
Hall-Cheetaly-
Ant-Pwd-Back
Discounl facter
0.99
0.99
0.99
0.9
Sample batch size
256
256
256
256
Task batch size
Real data ratio
0.5
0.5
0.5
0.5
Model rollout length
Inner critic Ir
le:3
le.3
8e-4
le.3
Inner actor Ir
le-3
5e-4
5e-4
1e-3
Inner steps
10
10
Outer cnitic
le-3
le-3
le-3
le-3
Outer actor
1o-3
1c-3
Meta-q It
le-3
le-3
le-3
le-3
Task
le.4
Metn-model Ir
Se-2
5e-2
Se-2
5e-2
Model uduptation sleps
25
Optimizer
Adam
Adam
Adam
Adam
Auto-tune
True
True
True
Tue
A.
À initial
5
100
100
Target divergence
0.05
Auto-tune
True
True
True
Trie
log
lo-3
lc-3
1e-3
Tog, initial
Q difference threshold
5
10
10
Max entropy
True
0.4
0.4
0.4
0.4
Testing adaptation steps
100
# training tasks
20
40
# testing tasks
Tune X. Similarly, we can optimize in policy improvement by solving the following problem:
max min (e 1Q" (s, a)] Aa(D(T, ma) Diarges] X(1 a Diarget]
Intuitively, the value of À will be increased to have stronger regularizations if the divergence is larger than some threshold Pranget- and decreased if the divergence is smaller than Dtarget-
Besides, we also build a meta-Q network Qmeta over the training process as an initialization of the task Q networks facilitate the within task policy optimization. At the k-th meta-iteration for meta-policy update, the meta-Q network is also updated using the average Q-values current batch B of training tasks with meta-q learning rate Ex. i.e.,
Q 1 Q E E
1
Σ Qnl.
B
NEB
Therefore, we initialize the task Q networks and the task policy with the meta-Q network and the meta-policy, respectively, for within task policy optimization during both meta-training and meta- testing. The hyperparameters used in evaluation of MerPO are listed Table
For evaluating the performance improvement in single new offline task, smaller learning rate Se for the Q network and the policy network update.
A.2.3 MORE EXPERIMENTS.
We also evaluate the impact of the utilization extent of the learnt model, by comparing the perfor- mance of MerPO under different cases of real data ratio, i.e., the ratio of the data from the offline dataset the data batch for training. As shown in Figure (a)] the performance of MerPO can be further boosted with a more conservative utilization of the model.
To understand how much benefit MerPO can bring for policy learning unseen offline RL tasks, we compare the performance of the following cases with respect to the gradient steps taken for learning in unseen offline RL tasks: (1) Initialize the task policy network with the meta-policy and run RAC ("With meta"); (2) Run RAC using the meta-policy without network initialization ("With meta (no mit))"); (3) Run RAC with single regularization based on behavior policy without network initialization. i..., 9(b]. "With meta" achieves the best performance and improves significantly over "No regul." and "With beha only", i.e., learning alone without any guidance of meta-policy, which implies that the
("With beha only"); (4) Run COMBO ("No regul."). As shown in Figure
14

---

Published as a conference paper at ICLR 2022
1e2
Welker-2D-Perams
Walker-2D-Params
Point-Robott-Wind
-4
-6
Average Return
Average Return
Average Return
-8
-10
-16
-18
0.0
1.0
2.0
3.0
4.0
5.0
0.0 1.0 2.0
3.0 4.0 5.0 6.0
0.0
0.2
0.5
0.8
1.0
Gradient Steps (x109)
Gradient Steps (x10*)
Gradient Steps (x10²)
cata ratio=0.99
beha only No regul.
Real cata retiomg.5
With meta (no Init)
300
300
(a) Impact of real data ratio.
(b) Performance comparison for (c) Training sample efficiency in Soint-Robot-Wind
unseen lasks.
Figure 9: Ablation study of MerPO.
1o2
Walker-2D-Paramn 2D-Params
1e2
Walker-2D- 2D-Parrm
4
Average Return
Average Return
3
0.0
1.0
2.0
3.0
4.0
5.0
0.0
0.2
0.4
0.6
0.8
1.0
Gradient Steps (x10*)
Gracient Steps (x10P)
5 trajectories
20 trajectoriss
trajectories
trajectories
Training efficiency.
(b) Testing efficiency.
Figure 10: Sample efficiency.
learnt meta-policy can efficiently guide the exploration out-of-distribution state-actions. Without network initialization, "With meta (no init)" and "With beha only" achieve similar performance because good offline dataset is considered here. Such result is also consistent with Figure 3(d)
We evaluate the testing performance of MerPO, by changing sample size of all tasks. Figure [O shows that the performance of McrPO stable even we decrease the number of trajectories for each task to be around 200. In contrast, the number of trajectories collected in other baselines is of the order 103. Figure 10(b) illustrates the testing sample efficiency of MerPO, by evaluating the performance at new offline tasks under different sample sizes. Clearly, good task-specific policy can be quickly adapted at a new task even with 5 trajectories (1000 samples) of offline data. We also evaluate the training sample efficiency of MerPO in Point-Robot Wind. As shown in Figure the performance of MerPO is stable even if we decrease the number of trajectories for each task to be around 200.
A.2.4 MORE COMPARISON BETWEEN FOCAL COMBO
Following the setup as Figure we compare the performance between FOCAL and COMBO in two more environments: Half- although FOCAL performs better than COMBO on the task with bad-quality dataset, is outper- formed by COMBO on the task with a good-quality dataset. This further confirms the observation made Figure T
and Ant- As shown in Figure [I]
A.3 ALGORITHMS
We include the details of MerPO in Algorithm
2
15

---

Published as a conference paper at ICLR 2022
1.2
Cheetsh (iled dehover
1.2
Cheetsh (Gzed detsser)
1c2. Art (tted datieee))
162 Art (Cened datasee)
Average Returs
1.0
1.0
Average Return
6.0
10.0
6.0
80 2.0
10.0
8,
Steps
Sieps
Gradient
Steps
COMBO
FOCAL
COMBO
FOCAL
(a) Performance comparison Half-Cheetah-Pwwl
(b) Performance comparison in Ant-Fwd-Bace
Back.
Figure 11: FOCAL COMBO.
Algorithm 2 Regularized policy optimization for model- based offline Meta-RL (MerPO)
1: Initialize the dynamics, actor and critic for each task, and initialize the meta-model and the meta-policy;
2: for = 1,.... do
3: for each training task M, do
4: Solve the following problem with gradient descent for h steps to compute the dynamics
model Tox based on the offline dataset D,:
min ((s,,,"") [log To, (s'1s, a)) + nl10, &m (k)112:
5:
end for
6: Update Φm.k + 1) = om. - E (k) E 0%]:
7: end for
8: Quickly obtain the estimated dynamics model T, for each training task by solving Eq. (4) with
steps gradient descent;
9: for 1, do
for each training task M, do for j = l.... do
11:
12:
Perform model rollouts with T, starting from states in D, and add model rollouts to
D
models
13:
Policy evaluation by recursively solving Eq. ( using data from D, U D Podeet Given the meta-policy = improve policy गे by solving Eq. 7
14:
15:
end for end for
16:
17:
Given the learnt policy b for each task, update the meta-policy *+1 by solving Eq. (10)
with step gradient descent;
18:
end for
B
PRELIMINARIES
For ease exposition, let TM and тM denote the dynamics and reward function the underlying MDP M, TM and "M denote the dynamics and reward function of the empirical MDP M induced by dataset D, and Tyı and r denote the dynamics and reward function of the learnt MDP M. To prevent any trivial bound with oo values, we assume that the cardinality of a state-action pair in dataset i.e., [D(s, a)], (]] in the denominator, non-zero, by setting [D(s, a) to be small value less than 1 when (s, a)
Following the same line as in Kumar et al. 2020 Yu et al. 2021b), we make the following standard assumption the concentration properties of the reward and dynamics for the empirical MDP M to characterize the sampling error.
Assumption 1. For any (s, E M, the following inequalities hold with probability 1 1-6: 8:
Or
Oro
| (T (s'sss) a)]l
a) (r)(), -mml=
ViD(s, a)l
where Cr,8 and Cr some constants depending 8 via vlog(1/5) dependency.
16

---

Published as a conference paper at ICLR 2022
Based on Assumption 1 we bound the estimation error induced by the empirical Bellman backup operator for any (s, a) E M:
X (s, - Q² (s. (s.a)
(m((,,0) 0) rm(s,a) +
<lrM(s, a) TM a)l +
[(Tu(o'(s,a)) - TM - (5',
Cr.
Tm(s'1a (s, ())1: 12t ('s','')]ll d')llloo
a)
Cr + yO Riman (1-7)
VID(s, a)
(1 Cr, Rmaa (CT,s)
(1 (D(,aa)
(Ors/Bmac + OT,5) Rmaz C...TBBaax
(1 (1-2) a)
(1 (1-7) (L-))//P(s.a)
Similarly, can bound the difference between the Bellman backup induced by the learnt MDP M and the underlying Bellman backup:
B 0" (s, ()
<|rM(s,a) rm) BBnaz (To M TM) 1-7
where (To M. TM) is the total-variation distance between TM and TM-
For any two MDPs, M₁ and .M2, with the same state space, action space and discount factor y and a given fraction f E (0, 1), define the f-interpolant MDP M, as the MDP with dynamics: TM, fTM, + (1 - f)TM, and reward function: TM, ffr, (1 f)r,, which has the same state space, action space and discount factor with Mi M2. Let 7 be the transition matrix on state-action pairs induced by stationary policy म, i.e.,
T = T(' (s, ()'(s'ss")
To prove the main result, we first restate the following lemma from (Yu al. 2021b) to be used later.
Lemma 1. For any policy n. its returns in any MDP M, denoted by J(M, T), and in M,, denoted by ((M1,Ma, 7), satisfy the following:
J(M - S M2,,, J(M, +
where
2y(1 f))
7f
17 (1 - 71- Rmax Deo (TM, Tm) + 1- Eng -7, Qiull
1-7 Es II (s,a) (rM..a)) 1 1-
f
Eg
as)
1-7
Lemma [ characterizes the relationship between policy returns in different MDPs in terms of the corresponding reward difference and dynamics difference.
C PROOF OF THEOREM
Let d(s,a) a) = ds, a). In the setting without function approximation, by setting the derivation of Equation Eq. (T) to O, we have that
gx+1(s,a) = 18x (t, gg(s,a) a) - d(s,a)
ds(s,a)
17

---

Published as a conference paper at ICLR 2022
Denote v(p. f) = Ep p(s.a)-d(s.a)
as the expected penalty on the Q-value. It can be shown Yu al. 2021b) that r(pp f) 2 0 and increases with f. for any p and (0, 1). Then, RAC optimizes the return of policy in f-interpolant MDP induced by the empirical MDP M and the learnt MDP M, which is regularized by both the behavior policy TH and the meta-policy Te:
(s,a)
u(0", .J) (m, mT) X(1 - D(T, me)-
max M,M
(12)
1-y
Denote as the solution to the above optimization problem. Based on Lemma | we can first characterize the return of the learnt policy To in the underlying MDP M in terms of return in the f-interpolant MDP:
J(M, To) + M,M
(13)
where
27(1 f)
Rman (Io TM) + 7f
- Eaaz [TM - TM Qiil
711
(1 - -1)
f
1-5 E Ba || ( rM
E Ea (s, -r)(s + 1-7
1 --7
27 - Roman (To M. TM)
(MT + I2SOT,slman E Bend (8)
|A
[(s) V Door (To (T) +1
(1 7)²
(1-
Cr,5 E Eg
+ 1-y
E, 1-y
Be (1, rm(ss
ViD(s,))
Note that the inequality above holds because the following is true for the empirical MDP M Kumaa al. 2020):
2Cr,BBmoz
|
Eay [TM 1T
Esnddy (s)
E
V (s))
+1
1-7
[D()]
for (TI = E. (c(ss)
(mielle) [z((a)))
1).
C.I SAFE IMPROVEMENT OVER Te
We first show that the learnt policy offers safe improvement over the meta-policy ne. Following the same line as in Eq. (I3), we next bound the return of the meta-policy т. in the underlying MDP M from above, in terms of its return in the f-interpolant MDP:
M,M, f, me) 2 J(M, Te) - 7/2
where
27 -
Rman (Io M TM) M. + 22, Rmar
IA
722 v
Bend)((s)
| [D())] V DooL (Te, (s) (Te) +
71-
21
+ Cr.6 E eeand
E andd ||r (1(1)),)))))) - as) (s)
viD(s,))
1-7
2ng.
It follows that
J(M, + v(0
XaD (To, TB) X(1 a)) (T me)
1-7
MJ - ((0=0,1)
((To Xa XaD T8) - X(1 (o: mc)
1-7
2J... - u(p
XaD(TTes R3)
1-9
ac) - -18 u(p==;1)
(T)
1-7
18

---

Published as a conference paper at ICLR 2022
where the second inequality is true because To is the solution to Eq. 12 This gives us lower bound on [M, To) in terms of (MM, Te):
B
ao) 2 JM, me) - 7f 112
1 1-9 [u(p" f]
+ (To TT) + X(1 - D) Te) Da TD)-
It is clear that vf and n/ are independent and X. To show the performance improvement of no over the meta-policy Te, it suffices to guarantee that for appropriate choices of / and A,
A. (T TB) A(1 ( Fe) (Te Ta) + 1 - [100 , u(p" []] 0.
To this end, the following lemma first provides an upper bound on ( v(p",,))
Lemma 2. There exist some positive constants L1 and L2 such that
|u(p" - u f)l s 2(L1 + L2) Dto (s,
Proof. First, we have that pupp"n .) u(p=";)))
p (s, d(s. a)
(= - d(s,a) (=
Fd(s,a) +(1 f) Epr
[fd(s,a) + +1 - f)
p a) d(s,a)
(s, p= a) - d(s,a) a) fd((, + (1 - f- f)
Σ
p (s,
fd(s, + (1- - (s,a)
(5.0)
( d(s, a) To (s, [d(s,a) + (1 - f0ptt (=c(ss,)) (s. - d(s, a) fd(3,a) + (1 - f) (8,
<
Σ
(8,a)
(=o a) d(s, a) p p= fd(ss (1 1) -='((s, fd((s,)) + (1 f) ( (s,
(s,a)
pTo d(s,a)
Σ
100 (s, (== a)l fel(s, (1 f)pto (8,
(8.a)
( d(s,a)
(x, - d(s,a)
+ Σ p ())
fd(ssa)+(11 - (s,)) fd(s, + (1- - f)p (s.
(s.a)
(0, 0) - -d(s,a)
<
(s, (s)
fd(s,a) (1 f)
(s,a)
( a) - d(s, a)
(s,0)) - d(3, a) a) + (1 f)- (8
+
p (s,
fd(s, + - fup (s,
(s,a)
(00
First, observe that for the term
+d(s.a)+(1- f) (s,a)
If (s, 2 ((s,a), then
pro a) d(s, a)
fd(s, a) + (1 - fo (s,
(s,
< ad(s + (1 - 10
<
(1 - (8,))
1--
If Te a) < d(s,a). a), then
(s, nd (8, a) - d(s,a) a)
d(s,a)
5 d(s,u) 1
5 fd(ss + (1 f)p (s,a)
fd(ss 0) + (1 f)p (s, (s
fd(s, a) f
19

---

Published as a conference paper at ICLR 2022
Therefore,
p (s, a) d(s.
1
S max 7
- l1-
fd(s,a) + (1 f00
Next, for the term (11.a))+(111)0011p))
('0(xx))d(x.a))
(=c(s,a) -c(ssa))d(saa)
consider the function g(ø)
[d(,,a)+(1- (101111-ppp 1)
a-d for E [0, 1]. Clearly, when d(s, = 0, fd+(I-fjr
pa a) d(s,a)
p² 0) d(s,a)
fd(s,a) + (1 (1- f),))s
fd(s,al a) + (1 - f)p= a)
For any (,,a) that d(ss a) > 0, it can be shown that g(x) is continuous and has bounded gradient,
i... Vg(x)) S P. - L2. Hence, it follows that pav (8,0) d(s,a) fd(s,a) + (1 (s,
(8, d(s. a)
S Lglp"" - p=c a)l.
Td(s,a)+(( f) (0000((,000
Therefore, we can conclude that pxpp0, f) ump" f)l
L1 Σ ( - (8, + L2 Σ
ne
-
(s,a)
(x.a)
<(I Tz2)
Σ (8, -pax (s)
-2( + L2) (s, 0) (s)))a))))
Recall that
(d) p==(s,a)==d (5)))
which denote the marginal state-action distributions by rolling out To and in the learnt model M, respectively. 2emma gives gives an upper bound on the difference between the expected penalties induced under To and Te. with regard to the difference between the marginal state-action distribu- tions. Next, we need characterize the relationship between the marginal state-action distribution difference and the corresponding policy distance, which is captured in the following lemma.
Let (TT max, Dao (TI (TIIITT2 denote the maximum total-variation distance be- tween two policies क and πτ. Then, we can have that
Lemma 3.
( (8, S
max Dro (To((T((ols)(((((())))
Proof. Note that
20
(s. S (1-7) Σ nd (s..))
It then suffices to bound the state-action marginal difference at time t. Since both state-action marginals here correspond rolling out T, and He in the same MDP M, based on Lemma B.1 and B.2 in (Janner et al. 2019), we can obtain that
D (p) (8, a.ll) (s,a))
< Drn (o) To (8) loT (=))()))) (s)) + max Du((To(ass](mm(aal))) a
mt Dro (To (al s) (alss) + max max
=(t + 1) max D (To (als))(
which indicates that
< (1-7)
Σ (1 + 1) max (al(to(aab))()((((a))))))
1 - max DIo (To(s)((s(ssss
20

---

Published as a conference paper at ICLR 2022
Building on Lemma and Lemma 3 can show that
2(L1 + L2)
(() - "(p"", ,
max Deo (eT(ta(alb)llaa((a)))
1 1-y
- Caxx Dro [
Let D(..)) max, Deo (-1(-))). is clear that for À 2 Ao where Ao
[1 (1-3111-a)) and a < 2.2
CB
A. =>aD(To T8) - X(1 a) Te) ATD TT) + 8
[upp" f) - v1p
1-9
=AaD(TT T3) (To me) ATD(TT T3) + X(1 20)
8
+ 1-9 [1pp" ff ,(p==,
8
21(1 - -A(1-aa)DD D Te) +
1-7 [u(p") - v(p"
CB
>1(1 2a) 2() Te)
D(To:: Te)
1 1-?
CB
=(A Ao) - D me)
Aa 101--(1 2xx)
D D( Te) > 0.
1-71
In nutshell, we can conclude that with probability 1 1
C3 -
J(M, nO) 2 J(M Fe) vi n +(A 21 -2xx) (TTo: me)
Ao -20)
(6)
(c) where (a) depends on d but independent to A, (b) is positive and increases with A, and (c) is positive. This implies that an appropriate choice of À will make term (b) large enough to counteract term (a) and lead the performance improvement over the meta-policy πe.
J(M, To) 2 JMM, Te) + &1
where 1 2
C.2 SAFE IMPROVEMENT OVER TB
Next, we show that the learnt policy To achieves safe improvement over the behavior policy тg. Based on Lemma we have
J2M1, S. f. TB) 2 J[M, TB) 733
where
(7 - f)
(1 (1- 7)2 Riar Dia (TM, Tm) + 75 |E x [T - IMi
1 o
f
e [| (5, - (s a)l + 1 1- E
+ - γ
1 --g soand
Fo (1, as)
M
2y - f)
Rmma ( (TrO TM) + 22
|
(1 7)2
(1 7)-
(0)
[D(s)
Cr,5
+ 1- γ
+ 1-
sanni -
vID(s,a)]
Ang
8
21

---

Published as a conference paper at ICLR 2022
Therefore, it follows that
J(M no) + n1 B 00000 (TD mp) - X(1 a:) No: Te)
1 1-7
M, "(00001) ,) Xa (To TT)) X(1 a)) To: Re)
1-7
M. 1. - u(p"
1(1 - (T),)
1 1-9
2J( - 1/3 - 3 u(p=0,11
X(1 - D(T)
--7
which indicates that with probability - Jo) 2 (M,, 7/ 1/3 +Aa To) + X(1 (To
Te) X(1
(TT, me)
1 f) x(p=,,f))
where 1/ is some constant that depends on 5 but is independent to and X.
To conclude, we can have that with probability 1-
J(M, 2 max [ax E1, J(M, + +
where
E1 -1f - 1/2 + (A A1 (TTo
To) + Ao - -2a) Ao01
CB
D(To, Te)
(14)
and
= - 3 + AaD(TTo +8)+ - (T D(To: TT)) - X(1 - D
(15)
+ B [u(p" ff) v(p"0,
(16)
Moreover, as we noted earlier, &1 > 0 for suitably selected and 3.2 For the term v(p=
u(p",, {f) in ε2 where (fp"; is defined as Ep- [0 (=aa)_(saa)) as noted in (Yu et al. 2021b). v(p*,, f) is expected to be smaller than v(p"., f) practical scenarios, due to the fact that the dynamics T learnt via supervised learning is close to the underlying dynamics TM on the states
dy(s,a)
visited by the behavior policy πT. This directly indicates that d (s, a) is close to (s, a) and pT* close to d(s a). In this case,
M
3lu(p=o v(p=0, 21(1 - D(To
(T. T8)
We can show that for 1 c,
3
(Te T8) As =>a TB) A(1 - a) ne) (T)
+ T- [v(p00 -(p"*,1)
=>aD(To (3) ) + (1 Te) - A(1 a)D T3) (ToD T.)
=X[(2 + a) To, kp)
(1 a) Te) (1 (Tees))
>) Tg) ((T., Te) D(Tc: [T)]
>0,
and AS increases with X, which implies that
To) 2 (M, & J (M, RT) - n/ -/3 +A J(M, RT)
for an appropriate choice of X.
22


Reviews:
[{"review_id": "_otufk5DyUw", "reviewer": "Reviewer_zWVS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a new model-based offline meta-rl algorithm under the actor-critic approach. It is a nice addition to existing algorithms for the study of meta-RL problem. The difficulty of balancing the exploration with meta-policy and exploitation with offline dataset is an interesting observation, and the proposed solution is a reasonable remedy.\n\nThe main idea and the corresponding algorithm are well presented. The paper starts with a clear motivation in the introduction. After explaining the background, it points out the limitation of applying online meta-RL approach to the offline setting in section 4.2.1, and proposes its solution to introduce another regularisation around behaviour policy. The full algorithm include many components, meta-learning for dynamics, model based policy optimisation, conservative Q-learning, regularisation with behaviour and meta-policy. I am glad that the authors explain each component clearly and how they interact with each other.\n\nThe solution to add a second regularisation to task-specific policy seems simple and intuitive, but it is nice to show the proposed method does improve the performance both theoretically and empirically. The main theory results look reasonable but I haven't checked the detailed derivation in the appendix.\nThe experiments results are convincing and the ablation study is very helpful to show the contribution of each component.\n\nI have a few questions about the algorithm design:\n\n- CQL already encourages the Q function, and consequently the learned policy, to stay close to the behaviour policy. Why do you think it's not enough and we need another explicit regularisation to pull the policy close to the behaviour policy?\n\n- Instead of regularising task-policy to behaviour policy in Eq 7, we can also consider regularising the meta-policy to behaviour policy in order to prevent the quality of the meta-policy to decrease. Could the authors comment on this option?\n\n- Selecting a proper alpha value is the key to the proposed algorithm. Unfortunately the range provided by Theorem 1 is not computable and it seems hard to find an appropriate value according to the available offline dataset of each new task in practice. While the authors show 0.4 is a good value for all the experiments, I doubt if we can trust it in general. Hoping a constant value of alpha to work for all is counterintuitive because the whole paper is arguing that we should balance the two regularisation according to the quality of the dataset and the meta-policy. So we should adjust the interpolation coefficient accordingly.\n\n\n\n\n\n\n\n", "summary_of_the_paper": "This paper considers the challenge in offline meta-RL problems, particularly, the difficulty with balancing a learned policy between exploring OOD state-actions by following the meta-policy and exploiting the offline dataset. To solve the problem, it starts with proximal meta-RL approach in the online setting, points out the problem of a degraded meta-policy in the offline setting due to lack of feedback from online interactions and proposes to regularize the task policy with additional penalty from deviating from the behaviour policy. It also integrates the model-based offline RL method for single tasks (COMBO) and offline meta-learning for task specific dynamics into its final form (MerPO). This paper provides theoretical analysis on its advantage over behaviour policy and the meta policy. Extensive experiments with ablation show the empirical performance meets its expected behaviour.", "summary_of_the_review": "The paper has a good motivation and provides a reasonable solution to the problem in offline meta-RL. Theoretical and empirical results support the advantage of the proposed algorithm over recent baseline algorithms.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WHNhGPw1eL6", "reviewer": "Reviewer_jFYQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Advantage:\nThe writing of the paper is clear and easy to follow. The proposed MerPO outperforms the state-of-art baselines and is intuitively explained well. The experiments show most of the performance of the components in the proposed algorithms, model-based vs model-free, involving behavior policy or not, the influence of the alpha, etc.\n\nDisadvantage/Problems under concern:\n1. About the important regularization parameter $\\alpha$. 1) Will an adaptive $\\alpha_t$ be better? Since the quality of the meta-policy $\\pi_c$ will be different during the learning process. 2) Do we need to use different alpha for each task when the behavior policy property of each task is different? This scenario seems didn't been discussed enough?\n2. How to choose alpha without information on the performance of the behavior policy and meta-policy? Do we need to try it out?\n3. There are some undefined or unclear notations, such as what is $\\rho_n$ in equation (5). \n\n\n", "summary_of_the_paper": "Targeting offline meta-reinforcement learning, the work proposes a model-based method called MerPO, with conservative value evaluation and individual policy improvement method with the tradeoff between the meta-policy and the behavior policy influence.\n\nThe main algorithm MerPO includes an initialization step and a two-loops meta-learning approach. The initialization step learns the model of the meta-model and dynamics for each task. With the fixed estimations of the models of tasks, the proposed merPO will alternatively update the policies for each task and the meta-policy.\n\nThe main contribution claimed in this paper is that the proposed method is a more robust method with the design of a regularization term involving the behavior policy, which improves the meta-learning policy when the behavior policy is actually better than the current meta-policy. Theoretical guarantees are displayed and experiments are conducted to show the performance of MerPO.", "summary_of_the_review": "The main contribution of this paper is well claimed and verified. With a small revision, the algorithm can achieve better performance on the examined tasks in this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MB_9hkUCG3j", "reviewer": "Reviewer_i86B", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is motivated by what the authors refer to as an inherent trade-off between exploiting the data in the offline dataset, and exploring out-of-distribution states (by means of a model). Empirically, they motivate this trade-off by noting that the quality of the data-set has a large impact on the efficacy of offline meta-RL approaches, and that single-task approaches to offline RL can be superior if data-quality is high.\n\nTo allow out-of-distribution exploration, they meta-learn an initialisation for a state-transition model via standard supervised meta-learning approaches. This initialisation is then used to rapidly adapt the model to a given dataset. The main part of the paper revolves around how to meta-learn a policy that can act as a regularizer in a policy-improvement step. They demonstrate empirically that if task-adaptation is regularised only towards the meta-learned policy, then performance can be surprisingly poor in the even that the meta-policy does not correspond to useful behaviour on the new task. Instead, the authors propose to regularize policy-improvement updates both towards the meta-learned policy and towards the behaviour policy implicitly defined in the offline dataset.\n\nWhile the authors make a good case for their design choices, my main issue with this paper is that it complicates the presentation unnecessarily, which obfuscates connections to prior works. As far as I can tell, the latter regularizer (towards the behaviour policy) corresponds to COMBO, but this is not made clear until several pages later in the experimental section. Hence, the gist of this paper is to extend COMBO to the multi-task setting by meta-learning, while also meta-learning a prior policy for proximal policy updates. This could be made much clearer, which would not only help place the contribution in proper context but also make the paper much more readable.\n\nA main strength of this paper is the careful motivation of each component of the proposed method. The authors demonstrate theoretically that both these regularisers are important for policy improvement, clearly motivating the need for a meta-learned prior as well as regularisation towards the dataset's behaviour policy. This observation holds also empirically, as the authors demonstrate that their method is at least as good as using only one of the two regularisers under various assumptions of data-quality. They also demonstrate that learning a model is critical for performance, and that their proposed method is better or on par with established offline meta-RL baselines.\n\nOverall, while I think the presentation of the method can be made much simpler and clearer, I believe this paper presents interesting findings for offline RL and has a strong proposal for an offline meta-RL algorithm.", "summary_of_the_paper": "This paper proposes a model-based meta-RL approach to offline learning over a distribution of MDPs (offline meta-RL). The proposed method is inspired by COMBO and in fact is a direct extension of it to the multi-task setting. The extension involves meta-learning a model-initialisation (or prior) that can be rapidly adapted to new offline RL tasks. They further add a proximal RL policy-improvement operator, where the prior policy is meta-learned over the task distribution. They establish theoretical guarantees for their method and conduct a careful empirical investigation to motivate its design, while establishing that it provides additional benefits compared to relevant baselines. ", "summary_of_the_review": "I recommend acceptance of this paper. Main strengths of the paper are:\n\n+ Compelling motivation of algorithm\n+ Careful analysis of its design\n+ Competitive performance\n\nMain weaknesses are\n- Overly complicated presentation of the algorithm\n\nPost rebuttal: \n\nI've read other reviews and the author's rebuttal and maintain my recommendation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4Cw6aqphYqq", "reviewer": "Reviewer_xD1s", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies an interesting question of offline meta RL and presents promising experimental results. The ablations do a good job of demonstrating that each component of the method contributes to the success of the overall method. \n\nOne concern with the paper is that the motivation for the method is rather unclear. This lack of clarity is due in part because the paper does not make specific, falsifiable claims. For example, one question posed is, “Why does the proximal Meta-RL method in Eq. (5) perform poorly in offline Meta-RL, even with conservative policy evaluation?” The paper continues to say that, “A poor meta-policy may have negative impact on the performance” and that, “following the meta-policy may lead to worse performance.” It is always possible that something “may” lead to worse performance, and this does not directly motivate the conclusion that, “it is necessary to balance the tradeoff between exploring with the meta-policy and exploiting the offline dataset, in order to guarantee the performance improvement of new offline tasks.” It would be great for the authors to provide direct evidence that this trade-off is necessary.\n\nSimilarly, the results in Figure 1 do not justify the claim that, “Clearly, existing offline Meta-RL fails to generalize equally well over datasets with varied quality.” Figure 1 only looks at one specific offline meta-RL algorithm and on one environment. Making such a general claim would require much more extensive evidence, and I do not see how, even if the claim were true, that would directly motivate the need to develop a method that will “strike the right balance between exploring with the meta-policy and exploiting the offline dataset.”\n\nIn short, the paper does not provide evidence that striking *this* balance is the main issue that must be addressed.\n\nAnother concern with the paper is that it is a relatively complex combination of existing components (COMBO, proximal meta RL, behavior cloning regularization), and so the onus is on the paper to demonstrate exceedingly good results to justify the complexity of the method. Although the results are positive, the method only mildly improves over FOCAL, and the paper would be strengthened by comparing to MACAW and BOReL, as those methods have been shown to perform well in these environments.\n\nAnother concern is that it is unclear how alpha is tuned or can be tuned in practice. The authors state that, “It is worth noting that different tasks can have different values of α to capture the heterogeneity of dataset qualities across tasks” but in practice, choosing a separate α for each task seems undesirable. If α was tuned for this method but no hyperparameters were tuned for baselines, this would also be concerning as this would bias the results in favor of MerPO.\n\nThere are some unjustified, or at least confusing claims, such as:\n\n“Because tasks are trained on offline datasets, value overestimation (Fujimoto et al., 2019) inevitably occurs in offline Meta-RL” Overestimation, as far as I know, is only an issue with value-based methods.\n\n“we study a more general offline Meta-RL problem.” I do not see how the problem statement in this paper is more general than the offline meta-RL problem present in past papers.\n\n“Learnt dynamics models not only serve as a natural remedy for task structure inference in offline Meta-RL, but also facilitate better exploration of out-of-distribution state-actions by generating synthetic rollouts” Evidence that learnt dynamics model help specifically because they facility better exploration is not provided.\n\n“Our results also provide a guidance for the algorithm design in terms of how to appropriately select the weights in the interpolation” I do not see how the theoretical results can practically guide appropriate choosing alpha. In the end, it seems like alpha still had to be chosen empirically.\n\nLastly, I have a clarification question: Can the method be applied to new, non-offline meta-RL tasks? One limitation of the approach is that RAC seems to depend on having a behavior policy to regularize against, making it impossible to use the resulting policy for online meta-RL.", "summary_of_the_paper": "The authors argue that offline meta-RL methods do not learn from poor data well, by demonstrating that COMBO (a single-task offline RL method) outperforms FOCAL (an offline meta-RL method) only when the training data is good. They use this to motivate a method called MerBO, which involves (1) learning a meta-dynamics model with proximal meta-RL and (2) updating a policy with real and synthetic data using a method called “RAC”, which is equivalent to COMBO but with an added KL regularizer against the behavior and meta-policy, and (3) updating the meta-policy used by RAC. The authors prove that under certain assumptions, the resulting policy will outperform both the meta-policy and the behavior policies. The authors compare the method to existing offline meta-RL methods and demonstrate that (1) the use of a dynamics model is important, (2) the addition of the behavior cloning regularizer is important, and (3) the use of the dynamics model is important to perform well on held-out offline RL tasks.", "summary_of_the_review": "The paper does not provide compelling evidence that their method addressed a critical problem, and the experiments make it difficult to know if the method proposed really outperforms current meta-offline RL method since details on tuning and important comparisons were not included. Moreover, the overall method is rather complex and potentially limited to the pure-offline RL (unless I am mistaken) evaluation scenario, making the practicality of the method questionable.\n\nEdit: Based on the rebuttal, I've increased my score up to a 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Summary**

This paper proposes a novel offline model-based meta-RL approach called MerPO. MerPO combines conservative value iteration with proximal RL policy iteration.  The proposed method is novel despite having some similarities to approaches like COMBO. The paper compares against it in the experiments. The paper provides both empirical and theoretical justification for the proposed approach.

**Final Thoughts**

Overall, I think the authors did a pretty good job at addressing the reviewers' concerns. Overall, I think this is an interesting contribution to the ICLR community. The reviewers were all positive about this paper. For the camera-ready version of the paper, I would recommend the authors to go over the reviewers' concerns again and make sure that those concerns are addressed in the paper too as they did in the rebuttal. Some captions are pretty short; for example, see the captions of figure 6 and figure 7. I would recommend the authors add more description to the captions in the camera-ready version of this paper.
Decision: Accept (Poster)