Correcting Data Distribution Mismatch in Offline Meta-Reinforcement Learning with Few-Shot Online Adaptation
Authors: Jianhao Wang
Jin Zhang
Haozhe Jiang
Junyu Zhang
Liwei Wang
Chongjie Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: offline meta reinforcement learning
offline reinforcement learning
meta-reinforcement learning
few-shot online adaptation
data distribution mismatch correction

Abstract: Offline meta-reinforcement learning (offline meta-RL) extracts knowledge from a given dataset of multiple tasks and achieves fast adaptation to new tasks. Recent offline meta-RL methods typically use task-dependent behavior policies (e.g., training RL agents on each individual task) to collect a multi-task dataset and learn an offline meta-policy. However, these methods always require extra information for fast adaptation, such as offline context for testing tasks or oracle reward functions. Offline meta-RL with few-shot online adaptation remains an open problem. In this paper, we first formally characterize a unique challenge under this setting: data distribution mismatch between offline training and online adaptation. This distribution mismatch may lead to unreliable offline policy evaluation and the regular adaptation methods of online meta-RL will suffer. To address this challenge, we introduce a novel mechanism of data distribution correction, which ensures the consistency between offline and online evaluation by filtering out out-of-distribution episodes in online adaptation. As few-shot out-of-distribution episodes usually have lower returns, we propose a Greedy Context-based data distribution Correction approach, called GCC, which greedily infers how to solve new tasks. GCC diversely samples “task hypotheses” from the current posterior belief and selects a greedy hypothesis with the highest return to update the task belief. Our method is the first to provide an effective online adaptation without additional information, and can be combined with off-the-shelf context-based offline meta-training algorithms. Empirical experiments show that GCC achieves state-of-the-art performance on the Meta-World ML1 benchmark compared to baselines with/without offline adaptation. 

Under review as a conference paper at ICLR 2023
CORRECTING DATA DISTRIBUTION MISMATCH IN OF- FLINE META- MIFAARREINFORCMMNT LEARNING WITH FEW- SHOT ONLINE ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Offline meta-reinforcemeen learning (offline meta-RL) extracts knowledge from a given dataset of multiple tasks and achieves fast adaptation to new tasks. Re- cent offline meta-RL methods typically use task-dependent behavior policies (e.g., training RL agents on each individual task) to collect a multi-task dataset and learn an offline meta-policy. However, these methods always require extra in- formation for fast adaptation, such as offline context for testing tasks or oracle reward functions. Offline meta-RI. with few-shot online adaptation remains an open problem. In this paper, we first formally characterize unique challenge under this setting: reward and transition distribution mismatch between offline training and online adaptation. This distribution mismatch may lead to unreliable offline policy evaluation and the regular adaptation methods of online meta- RL will suffer. To address this challenge, we introduce novel mechanism of data distribution correction, which ensures the consistency between offline and online evaluation by filtering out out-of-distribution episodes in online adaptation. As few-shot out-of-distributin
episodes usually have lower returns, we propose a Greedy Context-bant based data distribution Correction approach, called GCC, which greedily infers how to solve new tasks. GCC diversely samples "task hypotheses" from the current posterior belief and selects greedy hypotheses with higher return to update the task belief. Our method is the first to provide an effective online adaptation without additional information, and can be combined with off-the-she thelf context-based offline meta-training algorithms. Empirical experiments show that GCC achieves taahe-ff-the-att performance on the Meta- World ML1 benchmark compared to baselines with/without offline adaptation.
INTRODUCTION
Human intelligence capable learning wide variety of skills from past history and can adapt to new environments by transferring skills with limited experience. Current reinforcement learning (RL)
has surpassed human-leved performance (Mnih et al.) 2013 Silver et al.) 2017 Hafner et al. 2019) amount of experience. However, in many real-world applications, RI. encounters two major challenges: multi-task efficiency and costly online interactions. In multi-task settings, such as robotic manipulation locomotion (Yu et al. 2020b), agents are expected solve new tasks with few-shot adaptation with previously learned knowledge. Moreover, collecting sufficient exploratory interactions is usually expensive dangerous in robotics (Ratailov al. 2021) autonomous driving Yu et al. 2018), and healthcare (Gottesman et al. 2019). One popular paradigm for breaking this practical barrier offtine meta reinforcement learning (offline meta-RL; Li et al. 2020 Mitchell
but requires
et al.) 2021), which trains meta-RI, agent with pre-collected offline multi-task datasets and enables fast policy adaptation to unseen tasks.
Recent offline meta-RI. methods have been proposed to utilize multi-task dataset collected by task- dependent behavior policies (Li et al. 2020: Dorfman et al. 202I). They show promise by solving new tasks with few-shot adaptation. However, existing offline meta-RL approaches require additional information or assumptions for fast adaptation. For example, FOCAL (Li al. 2020) and MACAW (Mitchell et al. ()221) conduct offline adaptation using extra offline contexts for meta-testing tasks. BOReL (Doriman et al. 202I) and SMAC (Pong et al. (022) employ few-shot online adaptation, in which the former assumes known reward functions, and the latter focuses on offline meta-training

---

Under review as a conference paper at ICLR 2023
Offline meta-training
Meta-testing
Why am I failing?
Goal!
Gool! t
Data
Goelf esa
Goal! +
Failed!
Goal!
path
Data path 3
Sight range
right
rightrange
righe
Sight range
Datasel of task
Dataset task
Dataset lask
Adaptation episode
Adaptation episode
Figure 1: Illustration of data distribution mismatch between offline training and online adaptation.
with unsupervised online samples (without rewards). Therefore, achieving effective few-shot online adaptation purely relying on online experience remains an open problem for offline meta-RL.
One particular challenge of meta- RL compared to meta supervised learning is the need to learn how to explore in the testing environments (Finn & Levine 2019). In offline meta-RL, the gap of reward and transition distribution between offline training and online adaptation presents a unique conundrum for meta-policy learning, namely data distribution mismatch. Note that this data distribution mismatch differs distributional shift in offline single-task RI. (Levine et al. 2020), which refers to the shifts between offline policy learning and a behavior policy, not the gap between reward and transition distribution. As illustrated in Figure T when we collect offline dataset using the expert policy for each task, the robot will be meta-trained on all successful trajectories (+). The robot aims to fast adapt to new tasks during meta-testing. However, when it tries the middle path and hits stone, this failed adaptation trajectory (-) does not match the offiin training data distribution, which can lead to false task inference or adaptation. To formally characterize this phenomenon, we utilize the perspective of Bayesian RI. (BRL: Duff 2002 ZZintgraf et al.. 2019 Dorfman et al. 2021) that maintains task belief given the context history and learns a meta pootiy on belief states. Our theoretical analysis shows that tese-depnndnnt data collection may lead the agent to -ustr-ddistriuution belief states during meta-testing and results in an inconsistency between offline veea--plicyy evaluation and online adaptation evaluation. This contrasts with policy evaluation consistency in offline single- task RL (Fujimoto et al. 2019). To deal with this inconsistency, we can choose either to trust the offline dataset to trust new experience and continue online exploration. The latter may not be able to collect sufficient data in few-shot adaptation to learn a good policy only on online data. Therefore, adopt the former strategy and introduce mechanism of online data distribution correction that meta-policies with Thompson sampling (Strens 2000) can filter out out- --i-rii-iiiiittiinn episodes and provide theoretical consistency guarantee in policy evaluations.
To realize our theoretical implications in practical settings, we propose a context-based offline meta-RL algorithm with a novel online adaptation mechanism, called Greedy Context-based data distribution Correction (GCC). To align adaptation context with the meta-training distribution, GCC utilizes greedy task inference, which diversely samples "task hypotheses" and selects hypotheses with higher return to update task belief. In Figure I the robot can sample other "task hypotheses" (i.e., try other paths) and the failed adaptation trajectory (middle) will not be used for task inference due to out-of-distribun a delicate context mechanism achieve effective online adaptation for offline meta-Re RL and has the advantage of directly combining with off-the-shelf context-based offline meta-training algorithms.
with a lower return. To our best knowledge, our method is the first to design
Our main contribution is to formalize specific challenge (i.e., data distribution mismatch) in offline meta-RL with online adaptation and propose greedy context mechanism with theoretical motivation. We extensively evaluate the performance of GCC didactic problems proposed by prior work (Rakelly al., 2219 Zhang et al. [02I]) and Meta- World ML1 benchmark with 50 tasks (Yu et al.) 20206). In these didactic problems, GCC demonstrates that our context mechanism can accurately infer task identification, whereas the original online adaptation methods will suffer due out-of- distribution data. Empirical results on the more challenging Meta- World ML1 benchmark show that GCC significantly outperforms baselines with few-shot online adaptation, and achieves better or comparable performance than offline adaptation baselines with expert context.
NOTATIONS AND PRELIMINARIES
2.1
STANDARD META-RI
The goal of meta-RL RL (Finn et al. 2017 Rakelly et al. 2019) is to train a meta- meta-pplicy that can quickly adapt new tasks using N adaptation episodes. The standard meta-rl setting deals with distribution
2

---

Under review as a conference paper at ICLR 2023
((K) over MDPs, in which each task K; sampled from p(ı)) presents finite-horizon MDP Zintgraf et al. 2019 Yin & Wang 2021). K2 is defined by a tuple A, R, H, P R including state space S, action space A, reward space R, planning horizon H, transition function P (s' (s a), and reward function R (r|s, a). Denote K. is the space of task Kg-- In this paper, we assume dynamics function P and reward function R may vary across tasks and share common structure. The meta-RI. RI. algorithms repeatedly sample batches of tasks to train a meta-policy, In the meta-testing, agents aim to rapidly adapt a good policy for new tasks drawn from p(A)).
From a perspective of Bayesian RL (BRL; Ghavamzadeh et al. 2015). recent meta-RL methods (Zintgraf al. 2019) utilize a to formalize standard meta-RL. A BAMDP is a belief MDP (Kaelbling et al. 1998) of special POMDP:|sttom
Markov decision process (BAMDP: Duff 2002)
Partially Observable MDP (POMDP: Astrom 1965) whose unobserved state information presents unknown task identification in N adaptation episodes. BAMDPs are defined as tuple M+ (S+, A, R. H+ Po' P R+) Zintgraff et al. 2019). in which S+ S x B is the hyper-state space, B is the space of task belief's over meta-RL MDPS, A is the action space, R is the reward space, H+ = N x H is the planning horizon across adaptation episodes, P (s
is the initial hyper- state distribution representing task distribution p(x), p+ (s/+1/ ss .e. Te ) is the transition function, and It+ (ra | az) is the reward function. A meta-policy π (at [s[]
BAMDPs prescribes distribution over actions for each hyper-state o (st: be)- The objective of meta-RI. agents is to find meta-policy π. that maximizes expected return, i.c., online policy evaluation denoted by JMI (Tt). Formal descriptions are deferred to Appendix A.1..3 A.1.3
2.2 OFFLINE META-RL
In the offline meta-RL setting, meta-learne only has access to an offline multi-task dataset D and is not allowed to interact with the environment during meta-training (Li et al. (020). Recent offline meta-RL methods (Dorfman et al. 2021) always utilize cask-dependent behavior policies p(H((t), which represents the random variable of the behavior policy ja(ass) conditioned on the random variable of the task K. For brevity, we overload [u] = =((((x). Similar to related work on offline RL (Yin & Wang 2021), we assume that D+ consists of multiple i.i.d. trajectories that are collected by executing dusk-dependent policies [u] in M We define the reward and transition distribution of the task-dependent data collection by PM [M]] (Jin et al. 202ID. i.c., for each step in a trajectory,
PP Past,In [i]] (re, (141 8841 ,st,a)) &
[PPx, (re, (1418141 8141 |se, ue) par+ (sf |N4, p4) )] (1)
where the reward and transition distribution of data collection with in a task Hy is defined as
PR, (Te, St+1 |st, ae) R (ra (st, p (8c41 (sc,ae)
(2)
and pa+t (s!" IKc,, m2) denotes the probability of st when executing 14 in task K.. Note that the offline dataset D+ can be narrow and large amount of state-action pairs not covered. These unseen state-action pairs will be erroneously estimated to have unrealistic values, called a phenomenon of extrapolation error (Fujimoto et al. 2019). To overcome extrapolation error in offline RL, related works (Fujimoto et al., 2019) introduce batch-constrained RL, which restricts the action space in order force policy selection of an agent with respect to a given dataset. A policy 0 for all (s+ a) tuples that are not contained in D Offline RL (Liu et al. [2020] Chen Jiang 2019) approximates policy evaluation for a natch-constraiied policy x by sampling from an offline dataset D+ which is denoted by Jo- (I+) and called Approximate Dynamic Programming (ADP; Bertsekas & Tsitsiklis) 1995). During meta-testing, RL agents perform online adaptation using meta-policy π in new tasks drawn from meta-RL task distribution. The reward and transition distribution of data collection with π in M+ during adaptation is defined by
ot is defined to be cotth--onstraieed by D if x (a Is
P Past (Te, 8t+1 |st, ax)) = R* (rx (oslssa, at) p (set1 (et,a)):
where Rt, P are defined in M Detailed formulations are deferred to Appendix A.1.4
3 THEORY: DATA DISTRIBUTION MISMATCH CORRECTION
Consistency of training and testing conditions is an important principle for machine learning (Finn & Levinee 2019). Recently, offline meta-RL with des-ddepnndent behavior policies (Dorfman et al. 2021) faces a special challenge: the reward and transition distribution in offline training and online adaptation may not match. We first build a theory about data distribution mismatch to understand
3

---

Under review as a conference paper at ICLR 2023
this phenomenon. Our theoretical analysis is based on Bayesian RL (BRL: Zintgraf et al. 2019) and demonstrates that data distribution mismatch may lead to -usto--distriuution belief states during meta-testing and results in a large gap between online and offline policy evaluation. To address this challenge, we propose a new mechanism to correct data distribution, which transforms the BAMDPs (Duff 20022) enrich information about task-dependent behavior policies into overall beliefs. This transformed BAMDPs provide reliable policy evaluation by filtering out out-of-distribution episodes.
3.1 DATA DISTRIBUTION MISMATCH OFFLINE META-RL
We define data distribution mismatch in offline meta-RL as follows.
Definition 1 (Data Distribution Mismatch). BAMDP M for each task-dependent behavior policies [n] and batch-constrained meta-policy x² the data distribution mismatch between π and [m] is defined by that 3s Gt s.l,,
Par+ (st az) > and PMI (re, St+1 |st az) & PMIIl (m) (re, 81+1 |st at):
(4)
where Phr+ (st ,0t) is the probability of reaching tuple (st,a) while executing x+ in M+ (formal definition deferred to Appendix and PMr, PM+ [pa] are the reward and transition distribution of data collection with πt" ann [pl] defined in Eq. 7 and (3). respectively.
The data distribution induced by n+ and [n] mismatches when the reward and transition distribution of n and [n] differs in a tuple (st, ..).. in which the agent can reach this tuple by executing л in Mr,iie, part (st a:)) > O. Note that if x can reach a tuple (s* at), this tuple is guaranteed to be collected the offline dataset, i.e., i...p..t (s a+) > O, because batch-constrained policy n will not select an action outside of the dataset collected by [u]] as introduced in Section 2.2
Theorem 1. There exists a BAMDP M with task task-dependent behavior policies [u] such that, for any batch-constrained meta-policy n² the data distribution induced by π and [u] does not match.
To serve a concrete example, we construct an offline meta-RL setting shown in Figure 2 In this example, there are v meta-RL tasks K (K1....,Ky, ky and behavior policies { J.b. where 2 3. Each task K; has one state 8p, v actions, and horizon H For each task Ki, RL agents can receive reward performing action a, During adaptation, the RL agent can interact with the environment within v episodes. The task distribution is uniform, the behavior policy of task K, is μú, and each behavior policy 14 will perform a;. When any batch-constrained meta-policy - selects an action à on the initial state st. find that
r*1 (1))=1,"111(()))
80
80
y"" (a.)=1,r" "10 (a.)=0
pkk:) 3. (.:p(p: (ms)) = and (a
Figure 2: A concrete example, which has meta- RL tasks, one state, v actions, v behavior policies, horizon H 1 in episode, and adaptation episodes, where v 2 3.
1
I M (r $0 à
Part.(()) (r= | =1
(5)
in which there is the probability of 1 to sample corresponding testing task, whose reward function of whereas the reward the offline dataset collected by [n] is all 1.
Fact 1. With any probability 1 1- 5 E [0, 1). there exists a BAMDP M with task dese-depnndnnt behavior policies [n4 such that, for any batch- constrained meta veta-policy π the agent will visit out- distrribbbtii belief states during meta-testing due to data distribution mismatch.
In the example shown in Figure 2 an offline multi-task dataset D is drawn from the task lask-dependent data collection FM.jj-]- Since the reward of D is all 1, the belief states in D* has two types: (i) all task possible st and (ii) determining task i with receiving reward in action a,, For any batch- constrained meta-polly peolcy x selecting action â, on s/ during meta- meta-testing testing, there has probability 1 - 1 to receive reward 0 and the belief state will become "excluding task j", which is not contained in D+ with v 3. For any d E (0, 1]. let > J. with probability out-of-distribution belief states during adaptation. Note that, this phenomenon will directly lead to the following proposition since RL agents will visit -ittr--distribttion belief states during meta-testing in BAMDPs and induce unreliable offline policy evaluation.
8, the agent will visit
4

---

Under review as a conference paper at ICLR 2023
There exists BAMDP M with task-dependent behavior policies [p]] such that, for any bonch-constrained meta-policy x=, the gap of policy evaluation of Λt between offline meta- meta-training and online adaptation is at least == where H is the planning horizon of M+ 2.
Proposition 1.
In Figure 2] an offline dataset D+ only contains reward 1, thus for each batch-constrained meta- meta-policy rt, the offline evaluation of .+ in D+ is JD+ (x+)= H+ wH, defined in Section 2.2 The optimal meta-policy π+. in this example is to enumerate ".....; a, until the task identification inferred from the action with reward of 1. A meta-policy + needs explore in the testing environments and its online policy evaluation is Jari (xt**) = #+ The detailed proof is deferred to Appendix A.2 Thus, the evaluation gap between offline meta-training and online adaptation is
H
Ja++ (x*) - JD+
(T+) ---
JD+ (T+) Jar+ (s+
2
Proposition 1 suggests that offline policy evaluation can fail by visiting out- -fftoff-ittibutionn belief states. As few-shot online adaptation, we choose to trust the offline dataset and it is imperative to build connection between online and offline evaluation for effective offline meta-policy training.
3.2 DATA DISTRIBUTION CORRECTION MECHANISM
To correct data distribution mismatch, we define the transformed BAMDPs as follows.
Definition 2 (Transformed BAMDPs, informal). A transformed BAMDP is defined by new BAMDP M", whose belief by is about meta-Ro RL MDPs K and task desendepnndntt behavior policies [p]] The goal of meta-RL agents is find a meta-policy # (at [x] that maximizes online policy evaluation
JTT (a+))
where 8 (se, be) is hyper-state of M Denote the offline dataset M by D=
Transformed BAMDPs maintain an overall belief about the task and behavior policies given the current context history. Compared the original BAMDPs stated in Section 2.2 the transformed BAMDPs incorporate additional information about offline data collection into belicis. This transformation will introduce an extra condition to the online adaptation process, as indicated by the following fact.
Fact 2. For feasible Bayesian belief updating, transformed BAMDPs confine the agent in the in- distribution belief states during meta-testing.
During online adaptation, RL agents construct a hyper-state s, = (st, br) from the context history and perform meta-policy # (ag s ). The new belief b, accounts for the uncertainty of task MDPs and task dekendeennd behavior policies. In contrast with Fact transformed BAMDPs do not allow the
agent visit aut-of-distribution belief states with D Otherwise, the context history will conflict with the belief of behavior policies, i.c., RL agents cannot update beliefs b, when they have observed an event that they believe have probability zero. For example in Figure RL agents can take an action a1 and receive reward of 0 during meta-testing. After observing an action @1, the posterior belief will be p (K1. lr be) = 1, which is contradictory because the reward function of 01 is 1 in K1- To support feasible Bayesian belief updating, transformed BAMDPs require the RL agents to filter out -ustof-distribution episodes and induce the following theorem.
Theorem 2. In transformed BAMDP M for each mask-dependett behavior policy [p] and batch- constrained meta-polic policy क*, the data distribution induced by म and [μ] matches after filtering out out-of-distribution episodes in online adaptation. Besides, the policy evaluation of / in offline meta-training and online adaptation will be asymptotically consistent, as the offline dataset grows.
In Theorem 2 the proof of consistent policy evaluation is similar to that in offline single-task RL (Fujimoto et al. 2019), whose proof is deferred to Appendix AA.3 This theorem indicates that we can design delicate context mechanism correct data distribution and guarantee the final performance of online adaptation by maximizing the expected future return during offline meta-training. Moreover, we prove that meta-policies with Thompson sampling (Strens 2000) can filter out out-of-distribution episodes in M with high probability, as the offline dataset D grows (see Appendix A.5
GCC: GREEDY CONTEXT-BASE DATA DISTRIBUTION CORRECTION
In this section, we investigate scheme to address data distribution mismatch in offline meta-RL with few-shot online adaptation. Inspired by the theoretical results discussed in Section 3 we aim to distinguish whether adaptation episode is in the distribution of the given meta-training dataset,
5

---

Under review as a conference paper at ICLR 2023
then meta- policies with Thompson sampling (Strens] 2000) can utilize in-distribution context infer
how to solve meta-testing tasks. However, in practice, accurate -uttof-distribution quantification is challenging (Yu et al., 2021 Wang et al. 202T). For example, conduct popular empirical measure of offline RI., uncertainty estimation via an ensemble (Yu et al. 2020e Kidambi et al. 2020). and find that such quantification works poorly in the offline meta-RI. (see Section[5.I). To address this challenge, we introduce a novel Greedy Context-based data distribution Correction (GCC), which elaborates greedy online adaptation mechanism and can directly combine with off-the-shelf offline meta-RL algorithms. GCC consists of two main components: (i) an off-the-shelf context-based based offline meta- veta-training method which extracts -nowlleeewedge from a given multi-task dataset, and (ii) greedy context-based online adaptation that realizes selective context mechanism to infer how to solve meta-testing tasks. The whole algorithm is illustrated in Algorithm
4.1 CONTEXT-BASSE OFFLINE META-TRA TRAINING
To support effective offline meta-training, we employ state-of-the-and off-the-shelf context- based algorithm, i.e., FOCAL (Li et al., 2020), which follows the algorithmic framework of popular meta-RL approach, PEARL (Rakelly et al., 2019). In this meta-training paradigm, task identification is modeled by latent task embedding 2. GCC meta-train a context encoder q=]cc), policy T [aass, 2). and a value function Q(s,, a, z). where is the context information including states, actions, rewards, and next states. The encoder q infers a task belief about the latent task variable based on the received context. We use ((2) to denote the prior distribution for c = 0. The policy æ and value function
the latent task variable 2, in which the representation of can be end-to-end trained on the RL losses of π or Q. In addition to the gradient from т or Q, recent offline meta-RL (Li et al. 2020 Yuan & Lu 2022 also uses a contrastive loss to help the representation of distinguish different tasks. We argue that meta- vetaatrainnng the inference network within given dataset implicitly incorporates the information about offline data collection into the belief, as discussed in Definition 3 Formally, in the offline setting, the prior distribution q(=) can approximately present distribution of the latent task variable 2 over meta-training tasks, i.e., a sample drawn from q(>) is equivalent to sampling a task during meta-training and inferring E using the offline context from the offline dataset. During meta-testing, few-shot -fitt--iisttitttionn episodes may confuse the context encoder q( ((zcc) and RL agents need to use in-distribution context to provide reliable task beliefs.
conditioned
4.2 GREEDY CONTEXT- BASED META-TESTING
GCC is context-based meta-RL algorithm (Rakelly et al. 2019), whose adaptation protocol follows the framework of Thompson sampling (Steen)) 2000). The RL agent will iteratively update task belief by interacting with the environment and improve the meta-policy based on the "task hypothesis" sampled from the current beliefs. This adaptation paradigm is generalized from Bayesian inference (Thompson 1933) and has solid background in RL theory (Agrawal & Goyal 2012). To ensure feasible Bayesian belief updating, we adopt a heuristic of offline RL (Fujimoto et al. 2019). i.e.,
few-shot -ii-r---iitiibution episodes generated by an offline-learned meta-policy usually have lower returns since offline training paradigm does not well-optimize meta- meta-policies on out-of oot-off-af -istiitiibttion states. Its contrapositive statement is that episodes with higher returns have a higher probability of being in- in-distribution and the true return of episodes can be good -iitt---iitiiittion quantification. In this way, GCC contains two steps to perform a greedy posterior belief update at each iteration: (i) a diverse sampling of latent task variables, and (ii) a greedy context mechanism that selects task embedding with higher return to update the belief. For each iteration t, denote the current task belief' by br, the meta-testing task by Kteat, and the number of belief updating iterations by nar-
diverse sampling of latent task variables will generate nž candidates of the task embedding
provide various policies (a |s, sf) for the subsequent context selection mechanism. Due to the contrastive loss applied to the representation of = (Li et al.) 2020).
21, denoting by Z, = {=t] <=0
-i-1
a closer task embedding & may yield a more similar policy. Hence, to encourage the diversity of
policies. each candidate 2/ E Z, is designed by
=/ = arg max (in ZE
min ..3
min = = V'ckksnn
||a==b1,, and ž = { gi bt
Re-1
(7)
N==
in which of aims to maximize the minimum distance to the previous embeddings among n; samples. This greedy method is similar to Sarthest-Point Clustering (Gonzalez) 1985). a two-approximation algorithm for an harhard problem Minimax Facility Location (Fowler et 198T), which seeks a set of locations minimize the maximum distance from other facilities to the set.

---

Under review as a conference paper at ICLR 2023
A greedy selective context mechanism will select latent task variable 24 E 2, with higher return to update the task belief by. To evaluate the return of each 2j, GCC utilizes the policy π r(a [s 2f)
to draw ne episodes in Fitests denoting by E { approximated by sampling:
n--1 The policy evaluation of a can be
j=0
H Σ nk ( A==
(c))).
Ineese (zt) = Incese (m (a 2[) 2 Tri. (E)) = Tee 1=0
(8)
where Jnc. (E)) is the average return of episodes Ej and rh et
is the reward of k-th step
in an episode e, The task belief update in GCC consists of two phases: (i) an initial stage and (ii) an iterative optimization process. In the initial stage, GCC aims to find a reliable initial task inference 20 using a large amount of nğ diverse candidates, i.e., the initial context is Co arg ma {EE} IReen (Eo), maintaining the corresponding task embedding 20, and deriving the posterior belief bi g(= co). In the following iterations, GCC utilizes iterative optimization method to maximize final performance during few-shot online adaptation, i.e., when /. >> 1, let
nz = and if Frin (E²) > Treen (ct-1), we have C2 E and update the posterior belief b++1 = q(2l Usr Ce), otherwise C. = Ct-1 and keep the belief be+1 = b. The final policy (a ss 21) will depend on the optimal task embedding 2t with the highest return.
Algorithm GCC: Greedy Context- based data distribution Correction
1: Require: An offline multi-task dataset D+, meta-testing task Krest p(x)), the number of iterations Yar, ni. and a context-based offline meta-training algorithm A (i.e., FOCAL)
2: Randomly initialize context encoder q==le), policy π (als, z), and value function Q(s, a,
q. ग,, and Q given an algorithm A and dataset D Offfine meta- meta-training 4: Generate a prior task distribution (2) using the dataset D
3: Offline
b Start online meta-testing
5: Collect diverse adaptation episodes {Ej 1 "6-1
j=0 using q(2) and π in Kiest
6: Compute the greedy context Co. task embedding 2n. and posterior belief by
An initial stage
7: for = Tlat do
An iterative optimization process
8: Collect a diverse episode Ep using by and [aa)s, 2,) in Keest
9: Compute the greedy context e, and posterior belief b,41
10: Derive the final policy Tout (a [s, 22) with the optimal task embedding 2t
11:
Return: Tout
EXPERIMENTS
In this section, we first study didactic example to analyze the out-of-distributiin problem, and show how GCC alleviates this problem by its greedy selective mechanism. Then we conduct large-scale experiments on Meta-World ML et al.) (0200a). a popular meta-RL benchmark that consists of 50 robot arm manipulation task sets. Finally, we perform ablation studies to analyze GCC's sensitivity to hyper-parameter settings and dataset qualities. Following FOCAL (Li et al. 2020). we use expert-level datasets sampled by policies trained with SAC on the corresponding tasks. We compare against FOCAL (Li et al. 2020) and MACAW (Mitchell et al.) 2021), as well as their online adaptation variants. We also compare against BOReL (Dorfman et al.] [021]. an offline Meta-RL. algorithm build upon VariBAD (Zintgraf et al., 2019). For fair comparison, we evaluate variant of BOReL that does not utilize oracle reward functions, as introduced in the original paper{DorImas al. 202T). FOCAL is built upon PEARL (Rakelly et al., 2019) and uses contrastive losses to learn context embeddings, while MACAW is a MAML-based (Finn et al.) (2177) algorithm and incorporates AWR (Peng et al. 2019). Both FOCAL and MACAW are originally proposed for the offline adaptation settings (i.e., with expert context). For online adaptation, we use online experiences instead of expert contexts, and adopt the adaptation protocol of PEARL and MAML, respectively.
Evaluation results averaged over six random seeds, and variance is measured by 95% bootstrapped confidence interval. Detailed hyper-parametts settings are deferred to Appendix B
5.1
DIDACTIC EXAMPLE
We introduce Point-Robor Robot, a simple 2D navigation task set commonly used in meta-RL (Rakelly et al. 2019 Zhang al. 202T). Figure 3(a) illustrates the distribution mismatch between offline
7

---

Under review as a conference paper at ICLR 2023
Point-Robot
Average Return
-8
FOCAL
FOCAL with Context
Goal distr lbution
Gool distribution
for meta-training task Corresponding offiine dataset
Gool for meta training task
Corresponding offline daraset
20
Online trajs (fitered out)
ep sodes' trajs
20 30
Online trajs (selected)
Traj selected uncertainty estimation
lterations
(a)
(b)
(c)
Figure 3: (a) Illustration of data distribution mismatch between offline meta-training (blue) and online adaptation (green and red trajectories). (b) The agent fails to identify in-distribution trajectories via uncertainty estimation. Trajectories are colored with corresponding normalized uncertainty estimation. (c) Adaptation performance of GCC, FOCAL, FOCAL with expert context, and selecting trajectories with uncertainty estimation.
meta-training and online adaptation, as the dataset is collected by task-dependent behavior policies. As consequence, directly performing adaptation with the online collected trajectories lead poor adaptation performance (see FOCAL in Figure ((c)). Figure ((b) shows that existing methods that estimate uncertainty with ensembles (Kidambi al.]| 20200) cannot correctly detect OOD data, as there is a large error on uncertainty estimation. GCC fixes this problem by greedily selecting trajectories. As shown in Figure 3[a] the end of the initial stage, GCC only updates its belief with the red trajectory as it has the highest return. After the initial stage, GCC iteratively optimizes the posterior belief to get the final policy. As shown in Figure [ GCC achieves comparable performance FOCAL with expert context and significantly outperforms FOCAL with online adaptation. Visualization of GCC's and FOCAL's adaptation trajectories after the initial stage are deferred to Appendix [D Further analysis and visualization about the uncertainty estimation methods are deferred to Appendix
5.2 MAIN RESULTS
We evaluate on Meta- World MLI Yu et al. 2020a), a popular meta-RL benchmark that consists of 50 robot arm manipulation task sets. Each task set consists of 50 tasks with different goals. For each task set, we use 40 tasks as meta-training tasks, and remain the other 10 tasks as meta-testing tasks. As shown in Table I GCC significantly outperforms baselines under the online context setting. With expert contexts, FOCAL and MACAW both achieve reasonable performance. GCC achieves better or comparable performance to baselines with expert contexts, which implies that expert contexts may not be necessary for offline meta-RL. Under online contexts, FOCAL fails due the data distribution mismatch between offline training and online adaptation. MACAW has the ability of online fine-tuning as it based on MAML, but it also suffers from the distribution mismatch problem, and online fine-tuning can hardly improve its performance within a few adaptation episodes. BOReL fails on most of the tasks, as BOReL without oracle reward functions will also suffer from the distribution mismatch problem, which is consistent with the results in the original paper. We also test the uncertainty estimation method on several representative tasks, and it performs similarly to FOCAL. Detailed results are deferred to Table in Appendix E.
Table 1: Algorithms' normalized scores averaged over 50 Meta- World ML1 task sets. Scores are normalized by expert-level policy return.
FOCAL with Expert Context
MACAW with Expert Context
GCC
FOCAL
MACAW
BOReL
0.73 + 0.07 0.53 + 0.1 0.18 + 0.1 0.67 0.07
0.68 + 0.07
0.04 + 0.01
Table 2 shows algorithms' performance on 20 representative Meta-World ML1 task sets, as well rewrse-reward version of Point-Robot and Cheetah- Vel, which are popular meta-RL. tasks (Li et al.. 2020). GCC achieves remarkable performance in most tasks and may fail in some hard tasks as offline meta-training is difficult. We also find that GCC achieves better or comparable performance to baselines with expert contexts on 33 out of the 50 task sets. Detailed algorithm performance on all 50 tasks as well as comparison to baselines with expert contexts are deferred to Appendix E.2 We further perform ablation studies hyper-parameed Appendix F) Results demonstrate that GCC is generally robust to the choice of papperr-aramtterrs and performs well on medium-quality datasets.
settings and dataset qualities, and results are deferred to
8

---

Under review as a conference paper at ICLR 2023
Table Performance on example tasks, bunch of Meta- Woold ML1 tasks with normalized scores. For Wera-World tasks, "-V2" is omitted for brevity.
Example Env
GCC
FOCAL
MACAW
BOReL
Coffee- Coffee-Push
1.26 + 0.13
0,66 = 0.07
0.01 = 0.01
0.00 = 0.00
Faucet- Close
1.12 0.01
1.06 0.02
0.07 0.01
0.13 0.03
Faucet-Open
1.05 + 0.02
1.01 = 0.02
0.08 = 0.04
0.12 = 0.05
Door-Close
0.99 0.00
0.97 + 0.01
0.00 0.00
0.37 0.19
Drawer- Close
0.99 = 0.02
0.96 = 0.04
0.53 = 0.50
0.00 = 0.00
Door-Lock
0.97 - 0.01
0.90 = 0.02
0.25 = 0.11
0.14 = 0.00
Wate-SSide-Band Back
0.96 + 0.02
0.58 = 0.06
0.21 = 0.17
0.01 = 0.00
Dial-Turn
0.91 - 0.05
0.84 = 0.09
0.00 = 0.00
0.00 = 0.00
Handle-Press
0.88 0.05
0.87 + 0.02
0.28 0.10
0.01 0.00
Hammer
0.84 + 0.06
0.59 - 0.07
0.10 = 0.01
0.09 = 0.01
Button-Press
0.74 0.08
0.68 + 0.14
0.02 0.01
0.01 0.01
Push- Wall
0.71 + 0.15
0.43 = 0,06
0.23 = 0.18
0.00 = 0.00
Hand-Insert
0.63 0.04
0.29 = 0.07
0.02 = 0.01
0.00 0.00
Peg-Unppn Peg-UnplugSSiee
0,56 + 0.07
0.19 + 0.09
0.00 + 0.00
0.00 - 0.00
Bin-Picking
0.53 0.16
0.31 = 0.21
0.66 0.11
0.00 0.00
Soccer
0.44 + 0.04
0.11 = 0.03
0.38 = 0.31
0.04 = 0.02
Coffee-Pull
0.40 + 0.05
0.23 = 0.04
0.19 = 0.12
0.00 = 0.00
Pick-Place- Wall
0.28 0.12
0.09 = 0.04
0.39 = 0.25
0.00 0.00
Pick-Out-OP Ofut-Of-Hole
0.26 - 0.25
0.16 = 0.16
0.59 = 0.06
0.00 = 0.00
0.14 0.04
0.13 = 0.09
0.00 = 0.00
0.00 0.00
Cheetah- Vel
-171.5 22.00
-28.77 + 30.6
-23..0 23.5
-301.4 - 36.8
Point-Roke Roht
-5.10 = 0.26
-15.38 + 0.95
-14.61 + 0.98
-17.28 + 1.16
Point-Robot-Sparse
7.78 + 0.64
0.83 0.37
0.00 0.00
0.00 0.00
RELATED WORK
In the literature, offline meta-RI. methods utilize a context-based (Rakelly et al.
2019) or gradient-
based (Finn et al. 2017) meta-RL framework to solve new tasks with few-shot adaptation. They utilize the techniques of contrastive learning (Li et al.) 2020; Yuan & Lu 2022), more expressive power (Mitchell et al. 202T). or reward relabeling (Dorfman et al. 2021] Pong et al. 2022) with various popular offiin single-task RL tricks, i.e., using KL divergence (Wu al. 2019 Peng et al.
2019 Nair et al. 2020) or explicitly constraining the policy to be close to the dataset (Fujimoto et al.
2019 Zhou et al. 2020). However, these methods always require extra information for fast adaptation,
such as offline context for testing tasks (Li et al. 2020: Mitchell et al. [ Yuan & Lu 20222, oracle reward functions (Dorfman al. 2021). or free interactions without reward supervision (Pong et
2022). To address the challenge, we propose GCC, a greedy context mechanism with theoretical motivation, to perform effective online adaptation without requiring additional information.
The concepts of distribution shift in z-space in Pong et al. (2022) and MDP ambiguity in Dorfman et al. (202T) are related the data distribution mismatch proposed in this paper. We reveal that the task- dependent behavior policies will induce different reward and transition distribution between offline meta-training and online adaptation, which this mismatch combined with "policy" distribution shift in offline single-task RL (Levine et al. 2020) are two essential factors in the phenomenon of distribution shift in z-space (Pong et al. 2022). After filtering out these -ittf--distribttion data, GCC maintain overall belief about the task with behavior policies to address MDP ambiguity.
CONCLUSION
This paper formalizes data distribution mismatch in offline meta-RL with online adaptation and introduces GCC, novel context- based online adaptation approach. Inspired by theoretical implica- tions, GCC adopts greedy context mechanism to filter out out-of- out-of-diitrriution with lower return for online data correction. We demonstrate that GCC can perform accurate task inference and achieve thth--ff performance on Meta-World ML1 benchmark with 50 tasks. Compared to offline adaptation baselines with expert context, GCC also performs better or comparably, suggesting that offline context may not be necessary for the testing environments. One potential future direction is to extend GCC to gradient- based online adaptation methods with data distribution correction and to deal with task distributional shift problem in offline meta-RI. (which is not considered in GCC).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi- bandit problem. In Conference on learning theory, pp. 39-1. JMLR Workshop and Conference Proceedings, 2012.
Alex Slivkins Alckh Agarwal. Lecture 10: Reinforcement learning. In COMS E6998.001, Columbia University. 2017. OpenCourseeLecturee
Karl Astrom. Optimal control of markov decision processes with incomplete state estimation. Math. Anal. Applic., 10:174-206 1965.
Dimitri P Bertsekas and John N Tsitsiklis. Neuro-dynamic programming: overview. In Proceedings of 1995 34th IEEE conference on decision and control, volume 1, pp. 560-564. IEEE, 1995.
Anthony R Cassandra, Leslie Pack Kaelbling, and Michael L Littman. Acting optimally in partially observable stochastic domains. In Aaai, volume 94, pp. 1023-1028, 1994.
Jinglin Chen and Nan Jiang. Information-theeortiic considerations in batch reinforcement learning. In International Conference on Machine Learning, pp. 1042-1051. PMLR, 2019.
Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement -intninniddeniffabblityy challenges and effective data collection strategies. Advances Neural Information Processing Systems, 34, 2021.
Simon S Du, Sham M Kakade, Ruosong Wang, and Lin F Yang. Is good representation sufficient for sample efficient reinforcement learning? arXiv preprint 19XXiv:9110000 100003,,,, 2019.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning, arXiv preprint arXiv:1611.02779, 2016.
Michael O'Gordon Duff. Optimal Learning: Computational procedures for Bayes-adaptive Markov
decision processes. University of Massachusetts Amherst, 2002.
Chelsea Finn and Sergey Levine. Meta-leaming: from few-shot learning to rapid reinforcement learning. In /CMI., 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic weaanng for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Von Volume 70, pp. 1122-11533 135. JMLR. org, 2017.
Robert Fowler, Michael S Paterson, and Steven L Tanimoto. Optimal packing and covering plane up-complete. Information processing letters, 12(3):133-15 137, 1981.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2062. PMLR, 2019.
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar, al. Bayesian reinforcement learning: A survey. Foundations and Trends@ in Machine Learning, 8(5-6):359 483, 2015.
Teofilo F Gonzalez. Clustering to minimize the maximum intercluster distance. Theoretical computer science, 38:293-306 -069, 1985.
Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Doshi-Velez, and Leo Anthony Celi. Guidelines for reinforcement learning in healthcare. Nature medicine, 25(1):16-18, 2019.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. International Conference on Learning Representations, 2019.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline ri? In International Conference on Machine Learning, pp. 5084-5096. PMLR, 2021.
Leslie Pack Kaelbling, Michael Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-1344 1998.
10

---

Under review as a conference paper at ICLR 2023
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 11810-21823, 2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 179-1191, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01644 2020.
Lanqing Li, Rui Yang, and Dijun Luo. Focal: Efficient fully-offline meta-reiifforcemmnnn learning via distance metric learning and behavior regularization, arXiv preprint arXiv:20100.0// 2020.
Yao Liu, Adith Swaminathan, Alckh Agarwal, and Emma Brunskill. Provably good batch reinforce- ment learning without great exploration. arXiv preprint arXiv:2007.08202, 2020.
Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. In International Conference on Machine Learning, pp. 7780-7791. PMLR, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXi::2006.09359 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage- regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 1rXiv: 2019.
Vitchyr H Pong. Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine. Offline meta-reiiforremment learning with online sulf-supervision. In International Conference on Machine Learning, pp. 17811 1-8899.. PMLR, 2022.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pp. 1154-1168. PMLR, 2021.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcemmnt learning via probabilistic context variables. In International Conference Machine Learning, pp. 5331-5340, 2019.
Tongzheng Ren, Jialian Li, Bo Dai, Simon S Du, and Sujay Sanghavi. Nearly horizon-free offline reinforcement learning. Advances neural information processing systems, 34, 2021.
Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. arXiv preprint 13X1v:2202.13890, 2022.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-335 2017.
Richard D Smallwood and Edward J Sondik. The optimal control of partially observable markov processes over a finite horizon. Operations research. 21(5):1071-108 1973.
Malcolm Strens. A bayesian framework for reinforcement learning. In ICML, volume 2000, pp. 943-950, 2000.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.

---

Under review as a conference paper at ICLR 2023
Csaba Szepesvári. Lecture 17: Batch rl: Introduction discussion. In CMPUT 653: Theoretical Foundations of Reinforcement Learning. University of Alberta. 2022. OpenCourseLecturr..
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of samples. Biometrika, 25(3/4):25-2994, 1933.
Jane X Wang. Zeb Kurth Merth-Nelsonn, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint 1tt.::1611.00700 2016.
Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. Advances in Neural Information Processing Systems, 34:29420-29444
2021.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. 2019.
arXiv preprint arXiv:1911.11.
Ming Yin and Yu-Xiang Wang. Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems, 34, 2021.
Ming Yin, Yu Bai, and Yu- Yu-Xiang Wang. Near-optimal provable uniform convergence in offline policy evaluation for reinforcement learning. arXiv preprint arXiv:2007.03760, 2020.
Ming Yin, Yu Bai, and Yu-Xiang Wang. Near-optimal offline reinforcement learning via double variance reduction. Advances in neural information processing systems, 34, 2021.
Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving video database with scalable annotation tooling. arXiv preprint IrXiv:1805: 046677 2(5);6, 2018.
Tianhe Yu, Deirdre Quillen, Zhanpeng Hc, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pp. 1094-1100. PMLR, 2020a.
Tianhe Yu. Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learming, pp. 1094-1100, PMLR, 2020b.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 111141199144422 2020c.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint arXiv:2102.08363: 2021.
Haoqi Yuan and Zongqing Lu. Robust task representations for offline meta-reinfforcmmen learnin via contrastive learning. In International Conference on Machine Learning, pp. 25747-25759, PMLR, 2022.
Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen, Yingfeng Chen, Changjie Fan, and Chongjie Zhang Metacure: Meta reinforcement learning with empowerment-driven exploration. In International Conference on Machine Learning, pp. 26600-1610., PMLR, 2021.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforce- ment learning. arXiv preprint 1xXiv:2011.072111 2020.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sehastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for mayes-adaptive deep rl via meta-learning. In International Conference on Learning Representations, 2019.
12

---

Under review as a conference paper at ICLR 2023
A THEORY
A.1 BACKGROUND
Throughout this paper, for given non-negative integer N Z4, [N] denote the set [0, 10..,..., N - 1]. For any object that is a function of/distribution over S, S x A, S x Ax 8 S Ax R, will treat as vector whenever convenient.
A.1.1 FINITE-HORIZON SINGLE- TASK RL
In single-task RL, agent interacts with a Markov Decision Process (MDP) maximize its cumulative reward (Sutton Barto 2018). A finite-horizon MDP is defined as a tuple = (S,A, R. H, P. R) (Zintgraf et al. space, is the reward space, H E 24 is the planning horizon, S A - (S) is the transition function which takes state-action pair and returns distribution over states, and R S A - A (R) the reward distribution. In particular, we consider finite state, action, and reward spaces in the Without loss of generality,
2019; Du al. 2019). where S is the state space, A is the action
assume fixed initial state sc A policy ग S + A (A) prescribes a distribution over actions for each state. The policy æ induces a (random) H/hhrrizon trajectory す (50, dn, ro: 91,a1..... 1 -1, where ~ (su),, ~ R(su. ao). $1 P(su, ag), 01 ~ π (s1), etc. To streamline our analysis, for each h E [H] we use S₁, S S to denote the set of states h-th timestep. and we assume S₁ do not intersect with each other. To simplify notation, we assume the transition from any state in Sh and any action to the initial state 8o, i.e., E SH-1, a E A, we have P(sa|s, E [0, 1], Vt [H] almost surely. Denote the probability of TH:
theoretical analysis, i.e., |S| < oo, |A| < oo, [R]
We also assume
p ((TH)
((rls))) R(re(se,ad) ar) (E(M)
II
II P(sc+1)s,,0) xE(M-1]
For any policy n, we define value function V as: vh E [M], Vs Sn.
V= (8) E B P
Σ rh t=h
(10)
T(als)
Ep-R((l,,))
if = H 1,
aEA
Σ <als)
E [r]
Σ P(3's,aaV(a())
otherwise,
eEAA
8'ES.+1
and visitation distribution of æ is defined by px(+) (S) which is vh [ Vs E S..
if = 0 and 80+
H
(s)
Σ
(3) (((l3) P(s(3,ā),
h O,
(11)
0,
otherwise,
and E S, A.r E R,
p=(s, a) = (s) (aass and (=(s,ar)) = = T(als) R(r(s, a).
(12)
The expected total reward induced by policy n, i.e., the policy evaluation of म, is defined by
(T) Vn (5o) H Σ Pa (s. a) E,- [r]
(13)
.EA
The goal of RL is to find policy æ that maximizes its expected return J [(T).
the initial state is sampled from distribution Pt. Note this is equivalent assuming a fixed initial state 6g, by setting P(so. a) P, for all a E A and now our state 81 equivalent to the initial state in their assumption. "The transition from the state in S does not affect learning in the finite-horzoon MDP M.
Some papers
13

---

Under review as a conference paper at ICLR 2023
A.1.2 OFFLINE
SINGLE-TASE RL
We consider the offline finite-horizon single-task RL setting, that is, a learner only has access to { (i.e. |D] KH tuples) and is not allowed to interact with the environment for additional online explorations. The data can be collected through multi-source logging policies and denote the unknown behavior policy μ.. Similar with
dataset D consisting of K trajectories {st at x4) re(m]
related work (Ren et al. 2021 Yin et al.) [020 Yin & Wang/ [021] Yin et al. [02T] Shi et al. 2022). }
assume that D is collected through interacting K Li.d. episodes using policy /1 in M. Define the reward and transition distribution of data collection with ja in M by Pas (Jin et al. 2021 i.e,
vt [H] in each episode,
PM (rts S141 (st,a)) ae) RM (ra at) PM ($++1 (s),
(14)
where the action a, is drawn from a behavior policy μs. Denote a dataset collected following the i.i.d. data collecting process, i.e., D ~ (IP 11) is an i.i.d. dataset. Note that the offline dataset D can be narrow collected by some behavior policy μ and large amount of state-action pairs are not contained in D. These unseen acteon pairs will be erroneously estimated to have unrealistic values, called a phenomenon extrapolation error (Fujimoto et al. 2019). To overcome extrapolation error in policy learning of finite MDPs, Fujimoto et al. (Fujimoto al. 2019) introduces batch-constrained RI,, which restricts the action space in order to force policy selection of an agent with respect to a subset of the given data. Thus, define batch-constrained policy set is
rTP {m (alss) O whenever (s, & D),
(15)
where denoting (s,a) E D if there exists a trajectory containing (s, a) in the dataset D, and similarly for 8 E 'D, (s, a, 'D, or (s, s') E DD. The batch-constrained policy set ITo consists of the policies that for any state s observed in the dataset DD the agent will not select an action outside of the dataset. Thus, for any batch-constrained policy π E _I",, define the approximate value function VP: SS + estimated from D (Fujimoto et al. 2019 Liu et al. 2020) as: Vh E [H], Vs E S..
H Σ rt
VD (s) E.g
(16)
Σ ((al))
if H 1,
(17)
Σ
[r (+)))
otherwise,
which is called Approximate Dynamic Programming (ADP) (Bertsekas & Tsitsiklis 1995) and such methods take sampling data as input and approximate the value-function (L.iu et al. 2020 Chen & Jiang 2019). In addition, define the approximate policy evaluation of estimated from 2 as
Jo(T) = VVP(2p (8o).
(18)
The offline RL literature (Fujimoto et al. 2019 Liu et al. 2020 Chen & Jiang 2019 Kumar et al.)
2019 2020) aims to utilize approximate expected total reward (D(T) with various conservatism regularizations (i.e., policy constraints, policy penalty, uncertainty penalty, etc.) (Levine al. 2020 to find a good policy within a batch-constrained policy set ITP
Similar offline lnite-horizon single-task RL theory (Ren et al. 2021 Yin et al.) 2021 Shi et al. 2022), define
2021 Yin al. 2020
Yin & Wang
dMd min (p a) as, > 0, Vs S, E A).
(19)
which is the minimal visitation state-action distribution induced by the behavior policy μ in M and is an intrinsic quantity required by theoretical offline learning (Yin et al. 2020). Note that, different from recent offline episodic RL theory (Ren et al., 2021; Yin et al. 2020 Yin & Wang 2021: Yin et al.] |222 (hi et al. 2022). we do not assume any weak or uniform converage assumption in the dataset because we focus on the policy evaluation of all constrnnntt policies in II" rather than the optimal policy in the MDP M.
14

---

Under review as a conference paper at ICLR 2023
A.1.3 STANDARD META-RL
The goal of meta-RL (Finn et al. 2017 Rakelly et al.) 2019) is to train a meta-policy that can quickly adapt to tasks using N adaptation episodes. The standard meta-RL setting deals with distribution p(k) over MDPs, in which each task K; sampled from p(x) presents a finite- horiznn MDP (Zintgraf al. 2019 Du et al. 2019). K; is defined by a tuple (S,A, H, pk, R()) including state space S, action space A, reward space R. planning horizon H, transition function p ( (s'ls, a). and reward function R (rss, a... Denote K is the space of task Ką. In this paper, we assume dynamics function / and reward function R may vary across tasks and share common structure. The meta-RL algorithms repeatedly sample batches of tasks to train meta-policy. In the meta-testing, agents aim to rapidly adapt a good policy for new tasks drawn from p(k).
POMDPS. We can formalize the meta-RL with few-shot adaptation as a specific finite-horizon Partially Observable Markov Decision Process (POMDP), which defined by a tuple M = where S S K. is the state space, A and R are the tion and reward spaces as the umite-horizon MDP M defined in Appendix A.I.I respectively, 52 = S is the observation space, H N H is the planning horizon which represents N adaptation episodes for a single meta-RL MDP Ri, as discussed in Zintgraf et al. (Zintgraf et al. 2019).
ac-
P P:Sx. Sx A- A (8) the transition function: Vs, S, A, where denoting (s,Kg) and S' (s',Kz).
P ( (st(s's
if Kj, otherwise,
(20)
10
Po: (5) is the initial state distribution: VS (s, Kz) E S,
(p(Ks),
if 80: otherwise,
P. (8)
(21)
0,
0:S-A(0) is the observation probability distribution conditioned on state: V (s, K;) 8,0 12
f1 0,
1.
o
(ols)
(22)
otherwise,
and R A + A (R) is the reward distribution: VS (s, E A,E
(r(s, (rls,a).
(23)
Denote context Ca =
as an experience collected at timestep 6, and Ca (So. co...., Ct PE = 12xx (A x R x .2) indicates all experiences collected during t timesteps. Note that may be larger than H, and when is the case, Ca represents experiences collected across
Denote the entire context space C - C. and
episodes in the single meta-RL MDP meta-policy á ::C. - A (A) (Wang et al. 2016 Duan et al., (0016) prescribes a distribution over actions for each context. The goal of meta-RL is to find meta-policy on history contexts that maximizes the expected return within N adaptation episodes:
A Σ
(+)
E. E (onrrh:oroo(m/aaaatt/a/aaa---
rt
(24)
H-1
E (x(((())
E Pantt ). (4r (-)eee)))))
Σ
(25)
1=
For clarity, denote c. 80-
15

---

Under review as a conference paper at ICLR 2023
BAMDPs. A Markovian belief state allows POMDP to be formulated as Markov decision process where every belief is a state (Cassandra et al. 1994). We can transform the finite-horizon POMDP M to finite-horizon belief MDP, which is called Bayes-Adaptive MDP (BAMDP) in the
literature ZZingrraf et al., 2019 Ghavamzadeh et al. 2015; Dorfman et al. [021) and is defined
by tuple M (S* A. R. H P Po- R S
= S x B is the hyper-state space, where
B = (p(xle) le C} is the set of belief states over the meta-RL MDPs, the prior
by p (x/c,)) =p(x)
(26)
1]. Ve.((+1) C, denoting by = p (x(cx) and is the meta-RI. MDP distribution, and 14 E A 17-11
1 p (xlc)+)) = (x((c)) (((+1)) = (p (micae |ss, ct) = (b,
(27)
x pbb = p (cise, (1) (( (b/ - =p(cels bt) 68
(28)
- [ (ra]sr, at) p (st41/5,, at]] bi
(29)
is the posterior over the MDPs given the context C.(e+1)- 1). A, R are the same action space and reward
space as the finite-horizon POMDP M, respectively, H N x H the planning horizon across adaptation episodes. p+ : S x A x R + A (S*) is the transition function: Vs, st41 E S a E
Arr, E R, where denoting 55 (5t, bt) and 8/41 = (s4t..0/+))
pt (st-1 |st rt) = p+ 81+1, bitz 1s,, (e,,
(30)
pt (5c+1 |st, by, Tz) p (bi+1 |sc, (.
(31)
(s [P (st41/32, a:)l [b2z2 = p(b S es)
(32)
Po (S+ is the initial stper-state distribution, i.c., deterministic initial hyper-state is
+ = (80, 16) p(x)) E S+,
(33)
and R+ St x A - A (R) is the reward distribution: Vs+ = (5) E A,ERR, R+ (rls*, a) R (rls, 8 aa)= Ex [R (r(s,
(34)
In a BAMDP, the belief over the transition and reward functions, which are constant for given task. A meta-policy on BAMDP -+ S+ + A (AA) prescribes distribution over actions for each hyper-state. The agent's 's objective is now to find meta-policy on hyper-states π that maximizes the expected return in the BAMDP,
JM+ (T+ = E Ba an (o/o) Jor (+lod .s)) (ar.)) ) (- .s))
Σ
Tt
(35)
N-1
H-1 Σ
= E p(px))
Σ
BaNTT (115)44)) fee (14) (:1o),0))
T1
(36)
1=0
For any meta-policy on hyper-states - denote the corresponding meta-policy on history contexts
fxt C - A (A), i.e., VI E A -1] Ves E C., s.t., fa+ (-leze) = x (.1xt), where st =
(sc, =
and we have
N-1
JM (frt) ERjppp(x)
Σ
E
Baniet (f (1sss)))) (-see)))))))
Σ t=0
Tt
(37)
= JM (TT).
(38)
The belief MDP such that an optimal policy for it, coupled with the correct state estimator, will give rise to optimal behavior for the original POMDP (Astrom) 1965. Smallwood & Sondik 1973 Kaelbling et al. 1998), which indicates that
Jast (r+t*)= -x (fot..) JTT (#*),
(39)
where ti* and ** are the optimal policies for BAMDP M and POMDP M, respectively. Thus, the agent can find a policy n to maximize the expected return in the BAMDP M* to address the POMDP M by the transformed policy fat.
16

---

Under review as a conference paper at ICLR 2023
1.1.4 OFFLINE META-RI -L
In the offline meta-RL setting, meta-learner only has access to an offline multi-task dataset D and is not allowed to interact with the environment during meta-training (Li et al. 2020). Recent offline meta-RL methods (Doriman et al. 202I) always utilize task ass-dependent behavior policies p(u]k), which represents the random variable of the behavior policy (laass) conditioned on the random variable of the task K. For brevity, we overload [u] = p(u[(x) Similar to related work on offline RI. (Shi et al. (002)), we assume that D is collected through interacting multiple i.i.d. trajectories using task-dependent policies [μ] in M+ Define the reward and transition distribution of the task-dependen. data collection by PMI. .p] (Jin et al. 2021p, i.e., for each step in trajectory,
IP *M [o]] (Fa, 56++1 |st (e) x (Pxx)p(()) ((m((lss) [Px, (rt, 81+1 |8t, 0t) PPM (st |Na, (is:m))] (40)
where Fx, the reward and transition distribution of и; defined Eq. 14p, and PM+ (of [Ke me) denotes the probability of of when executing s4g in task Ks, i.e.,
PM+ (st (K).
Σ PH] 1 [b p(xle.)), C.EC
(41)
where the state in Ci. 8, and pk; (c:+) defined in Eq. Similar offline single-task RL (see Appendix A.1.2), offline dataset D be narrow and large amount of state-action pairs not contained. These unseen state-action pairs will be erroneously estimated to have unrealistic values, called a phenomenon extrapolation error (Fujimoto et al. 2019). To overcome extrapolation error in offline RI,, related works (Fujimoto et al.) 2019) introduce batch-constrained RL., which restricts the action space order force policy selection of an agent with respect to a given dataset. Define policy x to be batch-constrained by D* if x- (a Is
0 whenever tuple (s+ ,a) is not contained in D+. Offline RL (Liu et al. 2020 Chen Jiang 2019) approximates policy evaluation for a batch-consstriied policy x by sampling from an offline dataset D+, which is denoted by JD- (x*) and called Approximate Dynamic Programming (ADP; Bertsekas & Tsitsiklis) 1995). During meta-testing, RL agents perform online adaptation using meta- meta-policy in new tasks drawn from meta-RI. task distribution. The reward and transition distribution of data collection with π in M+ during adaptation is defined by
Par+ (re, Se+1 |stta, az) = RR (rx (t)) pt (stta|st,au)
(42)
where P- ($1+1 |st at) is the marginal transition functions of M i.c. P (Se+1 |st, (a) Bx (pms (8e41 (8t, at))
(43)
A.2 MAIN RESULTS IN SECTION
Definition 1 (Data Distribution Mismatch). In BAMDP M for each tesk-dependent behavior policies [p] and batch-constrained meta-policy n. the data distribution mismatch between nt and [m] is defined by that 3st Go s.t.,
Par (stt az) > 0 and Part (re.8e41 8641 (t)) # (Ms+ 8e41 |st at))
where Pir (st ae) is the probability of reaching tuple (st,a.) while executing x in M (formal definition deferred to Appendix A.2), and P M PM+ (us] are the reward and transition distribution of data collection with n and [u] defined in Eq. 77 and (3). respectively.
More specifically,
pxt+ at), = (xt+ + (at (8t))
(44)
where P (s is defined in Eq. D
Theorem There exists a BAMDP M+ with task- deskndependent behavior policies [j]] such that, for any batch-comstrained meta-policy x², the data distribution induced by t and [nu] does not match.
Proof. To serve a concrete example, we construct an offline meta-RL setting shown in Figure 4 In this example, there meta-RI. tasks K. (.i,..., K and behavior policies {pi,-- pu).
17

---

Under review as a conference paper at ICLR 2023
(01) (01)=1."" 1,0 (01)=0
S0
("((a))1,, (xp)) (a.)==
p(k.) = p(ul())) = 1, and (a(asss)) =
Figure 4: concrete example, which has meta-RL tasks, one state, actions, v behavior polices, horizon H = 1 in episode, and v adaptation episodes, where v 2 3.
where v 2 3. Each task K; has one state = {s0). , 2u actions A {ai,..., and horizon in an episode H 1. For each task Hg RL agents can receive reward performing action a, During adaptation, the RL agent can interact with the environment within v episodes. The task distribution is uniform, the behavior policy of task K; is Jú, and each behavior policy 14 will perform ag. When batch-constrained meta-policy πt selects an action à in the initial state si we find that
PM- (r = 50 à = PMrt [m] (r (r=11
+ a 11
(45)
which there the probability of + to sample corresponding testing task, whose reward function of ā is 1, whereas the reward in the offline dataset collected by [u] is all 1.
O
Fact 1. With any probability 1 ô E [0, 1), there exists a BAMDP M with task-dependent behavior policies [n] such that, for any batch- constrained meta- neta-policy nt the agent will visit out-of-distributin belief states during meta-testing due to data distribution mismatch.
Proof. In the example shown in Figure 4 an offline multi-task dataset D is drawn from the task- dependent data collection IM J[i Since the reward of D* is all the belief states in D has two types: (i) all task possible st and (ii) determining task with receiving reward in action ag. For any batch-constrained meta-policy π selecting an action ä on st during meta-testing, there has probability 1. receive reward 0 and the belief state will become "excluding task j", which is not contained in + with v 2 3. For any (0, 1), let > 1. with probability 8, the agent will visit ofstitribbutoo belief states during adaptation.
L
Proposition 1. There exists a BAMDP M with task-dependent behavior policies [j]] such that, for any batch-constrained meta-policy x², the gap of policy evaluation of n between offline meta-training and online adaptation is at least 1+-1 where H+ the planning horizon of M+
Proof. In the example shown in Figure 2 an offline dataset D+ only contains reward 1, thus for each natch-constrained meta-policy π². the offline evaluation of x in D* is JD+ (Λ+)= = H+ v.. The optimal meta-policy * in this example is to enumerate a1,..., G2 until the task identification inferred from action with reward of 1. A meta-policy +** needs explore in the testing environments and its online policy evaluation is
N-1 k-1
JMs (m-**)-
(46)
1=0
リーj-1
(47)
N(N - 1)
N
(48)
2v
K=D
0 e+1
H +1
(49)
2
2
18

---

Under review as a conference paper at ICLR 2023
where N = is the number of adaptation episodes. Thus, the gap of policy evaluation of x between offline meta- teta-training and online adaptation is
1Tst (-*) - Jo+ (x*)] | Jot (T+) Jnst (-+i-)= H
(50)
2
A.3 MAIN RESULTS SECTION 3.2
Definition 3 (Transformed BAMDPs (formal version of Definition 2 A transformed BAMDP is defined as tuple M (S A, p ,Po RR,R,, R. where S = S is the hyper-state
space, B is the space of overall beliefs over mela-RL. MDPs with behavior policies, A, R. H are the same action space, reward space, and planning horizon as the original BAMDP M+, respectively, POO is the initial hyper-state distribution presenting joint distribution of task and behavior policies p(s, p)) = (())). and P" R are the transition and reward functions. The goal of meta-RL. agents is find meta veeaa-poliyy a+ (as (5 that maximizes online policy evaluation I ( ***), Denote the reward and transition distribution of the task dependen data collection in a transformed
BAMDP M by F Iwl' as defined in Eq. (40). Denote the offline multi-task dataset collected by task- dependennddnt data collection P PM Am by D=
More specifically, finite-horizon transformed BAMDP is defined by a tuple MI = (s A,R, H+ P*, Pt R²)) R²), ) st x is the hyper-state space, where B = (c)
is the space of beliefs over meta-RL RL MDPs with behavior policies, the prior
F. = p (K. / (co) = p(k,
(51)
the distribution of meta-RL MDPs with behavior policies, and VI [ -1]. Vc,(++1) denoting br = (lca) and bit p (1s, p (((+))) = p (p (K. ploz)) (0+1) = p (x (x. plcce) 1st, = p(b 8t, ct)
C.
(52)
x <p (elss) 8() = p (ce |se, bith p (b = p (ce 1s,, by )bb
(53)
=B(R 114 (ar1s)) R (rels;, a.) pr (82+11 a;)] b5-11
(54)
is the posterior the meta-RL MDPs with behavior policies given the context C((++))- A, R and Nl are the same action space, reward space, and planning horizon as the finite-horizon BAMDP M respectively, p* :st A R
S is the transition function: vs, E S a
A,r r.r E R. where denoting , = (st, () and 8/41 = (SL+1. 87444
pº (st | (t,,)))))
P Sc41 Eual st, br. .as.r)
(55)
P
(8t+1 1st, brill rz) p+
st, br- ,c,)
(56)
(x)e)) 6f (soarr)) (8141 (s,, ar)]
1(bb cr)
p(b,
(57)
P+ A (st) is the initial hyper-state distribution, i.c., a deterministic initial hyper-state is
5 = (so, :) = (so. p(K E S*
(58)
and At A (R) is the reward distribution: Vs+ = (s, (s,t E tttaaEEEA. A.
R (r|st a) R² (r ls bm (oilla) a) E(re, [
(59)
19

---

Under review as a conference paper at ICLR 2023
In a transformed BAMDP M", the overall belief is about the task-dependent behavior policies, transition function, and reward function, which are constant for a given task. A meta-policy on M is actions for each hyper-state. With feasible Bayesian
++ : - A (AA) prescribes distribution belief updating, the objective of RL agents is now to find a meta-policy on hyper-states म that maximizes the expected return in the transformed BAMDP.
R-1 Σ T=
T
( = E ae (1/t )ouR ( (/ltt m))tim))
(60)
M
H-1
= pe ppp()
Σ
E
aca Ba 34 ) Dren (10e (1s
Σ r2
1=0
(61)
N-1
H-1 Σ r
= E (xp(e)
Σ j=0
Bar
aomaat (113 1e a1)+, (+18e,0))
(62)
For any meta-policy on hyper-states # denote the corresponding meta-policy on history contexts fat C A (A), i.., Vt ② -H-1 Vca+ Cc. s.t., fxt (-lc,) - (: ((()))
where 37
(st, b1 (st, (K, plc.)), and we have
N
H
Jis (fz1) = E (((e)
Σ
E Ba (f1ess in (. ~ PPni
Σ Tt
(63)
= -xa (#*).
(64)
Fact 2. For feasible Bayesian belief updating. transformed BAMDPs confine the agent in the in- distribution belief states during meta-testing.
Proof. During online adaptation, RL agents construct a hyper-state ड = (se. bi) from the context history and perform a meta-policy # (ar 5," ). The new belief b, accounts for the uncertainty of task MDPs and task-dependent behavior policies. In contrast with Fact [ for feasible Bayesian belief updating, transformed BAMDPs do not allow the agent to visit out- out-of-distributon belief states. Otherwise, the context history will conflict with the belief about behavior policies, i.e., RL agents cannot update their beliefs by when they have observed an event that they believe to have probability zero.
Lemma an MDP M, for each behavior policy P and batch-comstrained policy n, collect dataset 'D and the gap between approximate offline policy evaluation JD(TT) and accurate policy evaluation (M((F) will asymptotically approach to O, as the offline dataset D grows.
From a given dataset D, abstract MDP MD 2021 Szepesvári] (002). According to concentration bounds, the estimated transition and reward function will asymptotically approach M (Yin & Wang 2021) during the support of D. Then, using the simulation lemma (Alekh Agarwal (017:sseeesval 2022). the gap between JD((T) and IM (T) will asymptotically approach to as the offline dataset grows. Formal proofs are deferred in Appendix A.4
be estimated (Fujimoto et al.) 2019 Yin & Wang
Theorem 2. In a transformed BAMDP M for each task-dependent behavior policy [u] and batch- constrained meta-policy # the data distribution induced by - and [μ]] matches after filtering out ditttribitiiitooonn episodes in online adaptation. Besides, the policy evaluation of 7* in offline meta- meta-training and online adaptation will be asymptotically consistent, as the offline dataset grows.
Proof. We assume feasible Bayesian belief updating in this proof. At first, Vst ar s.t.
NT (st ae) > aim to prove
P t (re,8141 8/ at) = PMMt .u (re, at).
(65)
20

---

Under review as a conference paper at ICLR 2023
Since # is batch-constrained policy by [u], if p Mr
(st, ar) 0, we have plt M
(st at)
> 0. Then,
8841 8t41 PAy' (re,
,e) = R (ra 1st ai) P (s1+1 (t
(66)
Be (Prs (re, S1+1 Ist, (s,,aa)) at)
(67)
B [Px (Ta, 8241 (5c at
(68)
where 6, is defined in Eq. 51 and
p (Ra He (b = Ex (e [Eer-Par (eal1ss)))) [1 [6, pln, u[ca)]]
(69)
where pxrt (ca Mag pi) is defined in Eq. (9) According to Eq. 40 and (41).
P MI [al (Te, Sc-1 .c)
(70)
x p mx)
Prx, (res 8441 (s,, ac) . Px++ (st [ras 14)]
(71)
E Bert (calmase)) (Te (Fe, 8841 18,, (z) 16 p(s,
(72)
B [NN (Tr, 8841 (8e, at)) -Pa (rt, 8641 |8t ae))
(73)
(74)
Thus, the data distribution induced by # and [a] matches. Directly use Lemma a transformed BAMDP M in which M is belief MDP, a type of MDP. Therefore, the policy evaluation of # in offline meta-training and online adaptation will be asymptotically consistent, as the offline dataset grows.
A.4 OMITTED PROOF OF LEMMA I
Definition 4 (Dataset Induced Finite-Horizon MDPs). In a fmite-horizon MDP M with offline dataset D, dataset induced finite- horizon MDP is defined by M = (S,A, H, PM2 ,RM²) with the same state space, action space, reward space, and horizon as M. The transition function defined as follows: Vs, 8' SE
N(),a,8') a)
((, 0,
pMd (5'1s,0)=
(75)
0,
otherwise,
where N(s,a, a')) and N(s, a) are the number of times the tuples (s, a, s') and (s, a) are observed in D, respectively. The reward function defined by Vs ESS a E A,r E R,
N(s,
Nf, > 0.
A (rls,a) = (()
(76)
1o.
otherwise,
where (ss, r) is the number of times the tuple (s, a, are observed in D. The offline policy evaluation in D is equal to the policy evaluation in MD, i.e., for any batch-constrained policy x.
JD(TT) (T).
(77)
Note that dataset induced finie-horizznn MDPs M are not defined supports outside of dataset D. For simplicity, We set all undefined numbers to 0 in the transition and reward function.
Lemma 2 (Simulation Lemma for Offline Finite-Hon Finite-Horizon Horizon MDPs). In an MDP M with an offline dataset D. for any batch-constrained policy 11", if
max
M (-|8,a) a) - p.d (-18, a)]l Sep.
(78)
A with M (s,a>>0
ND ( a (x, Se.
(79)
A with (s.a)20
21

---

Under review as a conference paper at ICLR 2023
(s, max (r (s, a), M rM (s) mmax:
(80)
where r. (8, = (Ma)) and Mo () = PiwRMM F. we have
(T) [M)( HG,+ H(H - -Immax
cp-
(81)
Proof. Similar with famous Simulation Lemma in finite-horizon MDPs (Alekh Agarwal 2017). the proof is as follows. Recall value function wh E [H - 1], Vss E Sp (see Eq. (10),
(s) Σ T(als)
(s,
Σ
pM (f((
(82)
aEA
11
[M( = VAM (so), and vh E [H].
max
V (s) S (H-- h)r moa..
(83)
SES
We will prove wh E [M) Vs E Sn with PT Ma (s) > O,
(s) V V (s) (H (H h)e, + (H - h) - h - 1)raar
CP
(84)
2
by induction. When h = H 1, we have Vs E S₁. with p AN >0,
(s) - V (s) Σ (s,,a) - Σ
a(als)r (s,a) S
(85)
aEA
acA
holds. And 4h E [H - 11. aa E Sr with /
Mo (s) > 0,
MD (s) (s)
(86)
Σ T(uls) aEd
MD (s,a) +
Σ
pMn (111,
(87)
8'ESn43
Σ T(als) nr (s,
Σ (3((0,00)))31(
(88)
aEA
8'ES.11
T(lls)
- (s,
(89)
AEA
Σ T(als)
Σ
|p aV)MM (s"d)) (s)( (3)
(00)
aEd
S'ES,+1
SG +
Σ T(als)
(MD (s) a) Val (s))+
(91)
aEA
d'ESnt1
Σ (als))
MM (5 (8'1s,a) (311s,)))VMM((())
(5') - Va2 (s')
(92)
OEA
s'ES.+1
Sa. + (H -h++ 1+))))mmant
(93)
(H (h + ++))),+++ + (H - (h+ (H) (h+ + 1) 1)rmaz C.
(94)
2
(H h)c, + (H - (H - hh 1)rmaz Cp.
(95)
2
Thus,
ITMp (T) - [ns(TT] = VVMD (so) - V (so) S Her + H(H 1)rmaz
2 cp.
(96)
22

---

Under review as a conference paper at ICLR 2023
Lemma 3. In an MDP M with offline dataset DD collected by behavior policy ps, for any aatch-constraine policy x E II,, vg E (0, 11. with probability 1
log (1) + log 22 [A]
(T) - JM( & 11
(97)
Kd
where the number of trajectories in the dataset D and di the minimal visitation state-action distribution induced by the behavior policy μ in MD (see Eq. 19)).
Proof. Vs, E S,a E A with = a) > 0, note that P
2 d and according to (s, (
MD
Binomial theorem and Hoeffding's inequality, Ve E [0, 1],
P P (/|ssa) pM (s'ls,,)) 2 c) s 2
d + dx exp (-2c²))
(98)
2 (1
dx p 2) K
(99)
and
P P(r (s, (s ~ (s.a) a) 2 2() < 2 - ())
(100)
Thus, using union bound, Ag E (0, 1}, with probability 1 - 8, denote
max A with MD (s.a)>0
M (-1s,a) - pal (-18, ||
(101)
max =ES,aE with pt (s,a)
|8| pMM (11s, pMd (-1s, a))s
(102)
log (+) + log
22 (A)
|
(103)
K d Mo
log (1) + log (2 [S| [A)
max 6 sES aEA with An (s,a)>0
p Mo a) p (s)
(104)
K Kd
and mmaz = 1, thus, according to Lemma (a varianted simulation lemma in offline RL), we have
log (7) log (2|5² IA))
H H(H - 1) (s))
(T) - JM((F)
(105)
2
Kd
log (?) log (2|5)²1 IA))
S 111
(106)
K Kdjí
Lemma an MDP M, for each behavior policy P and batch--omstrrined policy n, collect dataset 'D and the gap between approximate offline policy evaluation TD(1 (T) and accurate policy evaluation TM(Tx) will asymptotically approach as the offline dataset 'D grows.
Proof. From Lemma as the size of an offline dataset [D] KH grows, the gap between approx- imate offline policy evaluation Tp (T) and accurate policy evaluation Jas (т) will asymptotically approach to O.
A.5 HOw TO FILTER OUT OUT-OFF-IITTRIUTTON EPISODES IN TRANSFORMED BAMDPS DURING META-TESTING?
Definition 5 (Sub-Datasets Collected by Single Task Data Collection). In a transformed BAMDP M an offline multi-task dataset D* is drawn from the task dask-dependent data collection PTT (n] A sub-dataset collected by behavior policy μ4 in task K; is defined by Daypu Note offline multi-task dataset is the union of sub-datasets D Driguo Le,
D*
U DRIpp
(107)
23

---

Under review as a conference paper at ICLR 2023
For each sub-dataset Dr. define batch-constrained policy set in single-task (K4,, (4)) as 1P (see the definition in Eq. (15)).
Definition 6 (Meta-Policy with Thompson Sampling). For each transformed BAMDP M a meta- policy set with Thompson sampling on M is defined by म न 8 B K. x [1 + A (A), where B is the space of beliefs over meta-RL MDPs with behavior policies, IC is the space of task R. and [ps is the space of task dependepn behavior policies. In each episode, म*. samples task hypothesis (Kq, p4) from the current belief byth, where t' is the starting step in this episode. During this episode, . (-1st, by R4g prescribes a distribution over actions for each state St, belief byy and task hypothesis (K4, Beliefs byd and task hypotheses (K4,4) will periodically update after each episode.
In the dlee-learming-baae based implementation, a context-based meta-RI. algorithm, PEARL (Rakelly) jtt al., 2209)) utilizes a meta-policy with Thompson sampling (Strens) 2000) to iteratively update task belief by interacting with the environment and improve the meta-policy based on the "task hypothesis" sampled from the current beliefs. We can adopt such adaptation protocol to design practical offline meta-RL algorithms for transformed BAMDPs.
Definition (Batch-Constrained Meta-Policy Set with Thompson Sampling). For each transformed BAMOP M with an offline multi-task dataset D batch--onstrained meta-policy set with Thomp- son sampling is defined by
T - + (ot 1st, by Ki, Ks, P4)
whenever (8t, 0t) & D Drippe Vo }
(108)
where denoting (s,, az) Drajus if there exists a trajectory containing (st, (8),a;) a.) in the dataset D Des.pe
The batch-constrained meta-policy set with Thompson sampling II consists of the meta- meta-policies that for any state 8, observed in the hypothesis dataset Dx the agent will not select an action outside of the dataset. Note that in each episode with task hypothesis (Ki, /4), a batch-constrained meta-policy with Thompson sampling .t. is batch-constrained within sub-dataset D Dr.,pee. i.c.,
Vot." we have म + (-1st, brt Ki, p4) ) E II
Definition 8 (Probability that a Policy Leaves the Dataset). In an MDP M and an arbitrary offline dataset D. for each policy : S + A (A)), the probability that executing i in M leaves the dataset
DD for an episode is defined by
M.D Pout
(TT)
Σ
P
M (TH)
[TH leaves D]
(109)
TH
Σ
M Pa (TH) 1 E [H] s.t. St & D or (s,, a, r() $ D),
(110)
TH
where pMd (TH) is the probability of executing in M to generate an H-horizon trajectory тh (see the definition in Eq. (9)). denoting &t E DD if there exists a trajectory containing st in the dataset D, and similarly for (81, rz) 'D
When we aim to confine the agent in the in-distribution states with high probability as the offline dataset grows, it is equivalent to bound the probability that executing a policy x in M leaves the dataset for an episode, i.e., M.D (T).
Theorem In a transformed BAMDP M with an offline multi-task dataset D' collected by dase-dependent behavior policies [u]], consider each batch-constrained meta-policy with Thompson sampling . E -TOt. .T (see Definition 6 and 7). For each adaptation episode in a meta-testing task Kitest p(K), denote the current belief by byrt there exists task hypothesis (Ra, ps) from by.". executing # # with (b%," Ris Ha) in Brest will confine the agent in the in-distribution belief states with high probability, as the offline dataset D grows.
From Theorem 3 for each adaptation episode in Ktest with the current belief 155."" we can sample task task hypothesis (Ki, (u) ~ b6. and execute म t.T with (by Kr,, pı) to interact with the environment until finding in-distribution episode. Thus, prove that peta-policies with Thompson sampling
24

---

Under review as a conference paper at ICLR 2023
can filter out out-of-distribution episodes in M with high probability, the offline dataset D grows. Note that Theorem B considers arbitrary task distribution p(x), since the distance between the closest meta- training task K₁- and Keest will asymptotically approach zero with high probability, as the i..dd offline meta-training tasks Ktrain sampled from p(k) in D grows. The detailed proofs are as follows.
Proof. For each batch-constrained meta-policy with Thompson sampling 7t E 1 ,I with (bg" KK, 14), similar to Definition 8. define the probability that executing x #t (-|se, by Ka, Mi) leaves the dataset D in an adaptation episode of meta-testing task Kinst ~ p(K):
Steat. D'' (7+ byt Ri, 1s)
Mi Kis 14) 1 TH leaves D'], Σ Pet (TH
Pout
(111)
where Pat. (TH byth Ras pi) is the probability of executing म t.T (-1se, bithh (i, in Reest to generate an H-horizon trajectory + in an adaptation episode, i.c..
P
(Til
(112)
1++H-1
1+
#+. (at (sc, by KirMi) Purrest (relse, ae)
II t=
precet (86+1 [s, (T)
(113)
t=
Pa Pa
(TH by ps),
(114)
where we can transform F to TH with the same probability since the belief 65t and task hypothsis (Ki, pi) will periodically update after each episode. Therefore,
Pout
D (#+iT by Ki, Mi)
Σ
Fiest (TH) by K4, p4) 1 (+ leaves D+1
(115)
Meast (THl by Kes p4) 1 [TH leaves D Dai- Σ
(116)
TH
Pout
Day
( 67 k...
(117)
Dage (5+ 1..TT bitl Ki4, (4) is the probability that executing . (-1s b5t Kg, (4) in Ktest leaves the sub-dataset Dr,- Me for an episode (see Definition [). Dx,- Me is a sub-dataset collected in Dt (see Definition 5) and K;- the closest offline meta-training task to Kitest, i.e.,
where Pout
Fig
arg min ||NG 1 -ktetlloo
(118)
H, Eeraen
arg min max (I) a, prico (s,a, s')l 11 (s, RNi HM, (s,
(119)
in which denoting the i.i.d. offline meta-training tasks sampled from p(ts) in D+ by Karain- From Lemma as the offline dataset D* grows, Drier sti.. and Kirain grow monotonically, for any muuch-constraied policy æ in Dr ie.. VT 11 wur, when exccuting æ in an episode of Deeas, Dagei (T), which asymptotically Netest p(k), the probability leaving the dataset D' is Pout approaches zero.
For the first adaptation episode a meta-testing task Kiest p(x) with the prior belief p(s, 1)) p(K))(u(()) there exists a task hypothesis (Ka mz-) in the prior p(s, n). then due to t ((,,, H)), kr.pp)) : 11 Pece from Definition and
Dt (5t.T ,p(ss, p)), (4.)) S Freat
Newn Deje ( ( ( p(n, p), Ky, K.-)) Paut
Pout
(120)
as the offline dataset Dt grows, executing म +.7 with (p(n, 4i), Ker, . for the first episode in Kiestt will confine the agent in in-distribution belief states with high probability. In the subsequent
25

---

Under review as a conference paper at ICLR 2023
adaptation episodes with current belief by in Kteat, the task hypothesis , is also in the belief byy by induction.
Therefore, for each adaptation episode with current belief b in Keest, there exists task hypothesis (Ki, (4) from t4 e.g.. (Ke ( executing * with (bt Ki, (s) in Ktest will confine the agent in in-distribution belief states with high probability, as the offline dataset D grows.
A.6 OMITTED LEMMAS FOR THEOREM 3
Lemma 4, In an MDP M and an arbitrary offline dataset 'D for each policy n, Ar (s, a, r)
M.D Pout (T) & H (11p (s) - (s)]l, + || (s, (s, P
(121)
where ( as, is the visitation distribution of (s, r) in M, defined in Eq. 12
Proof. For each TH leaving D, we use the first outlier data (s, & D or at, & D) to represent TH- Thus,
out (x)= = =-pt PT (TH) [3t E [M] s.t. 8: & DD or (82, az rz) & D
(122)
TH
S H
p (s) +
Σ sES,dEA,rER with MD (s.a.))=0 P (s, r)))
Pr (s.
M
(123)
(sES with Pm (=)=0
S H (s) Ma (s) + 11005 (s,a,r)
(124)
Lemma 5. In transformed BAMOP Mtt with an offline multi-task dataset Dt' collected by task- dependent behavior policies [p)) denoting the i.i.d. offline meta-traine traaairaiiing tasks sampled from p(k) in D by Ktrain for each meta-testing task Krest p(x), and denoting the closest offline meta-training task to Keest by Kp, i.e.,
Kiz arg min ||AG - -Ktestlloo
(125)
Kivain
arg min max (s, (s, 5') 11 (s,,,r) (0 Rii (s,
18"h
(126)
Hi is sub-dataset collected in then for any catchh-cossrraied policy x in D pp where Dr.- p.- (see Definition 5 i.e., Vn 11 1n
Ntest Papiies (T) Pout
(127)
log (+) + log (2|8(² IA)
log (1)
(ााालााा
< 2HH [
2
(128)
Kr Stg Mongean
[Kerain]
Asymptotically approuches zero.
Asymptotically approaches zero. when Keratn is sufficiently large
when Dage sufficiently large
MDx ar-
where Kr su.- is the number of trajectories in the sub-dataset Dr. sti dy. visitation state-action distribution induced by the behavior policy P. in MD. ir
is the minimal (see Eg. 179 ) and
[Ktrain is the number of i..dd offline meta-training tasks sampled from p(k) in D
Proof. According Lemma
Krest Dage (TT)
Pout
(129)
26

---

Under review as a conference paper at ICLR 2023
MoGuen (s) - D (s)
Monyriee (s,0,r) - P (s, (s,
S H
PT
+ PT
(130)
Mi (8) P (s) +
Mompiee (s, PN (s, ( (s, r1))
H
Pr
Px
(131)
H (1 f (s)
P f (s)]l + |1 (5, - P Mo (s,
(132)
Part I
Similar with Lemma 3 vg (0, 11. with probability 8, denote
pM (-15,a) pM: (1ss,))))
Cp
max Mp
(133)
aES,aEA Pr
(8,0)20
log (3) log (2|5]² (A1)
< $181,
(134)
MDAca.ie Kxpe-Hi dp
max #E8, a.A with Ma (s,0)20
MD (s - - (s
(135)
log (1) + log (2 (A))
(136)
MD
Keprpe du
thus, from Lemma 6 we have
MD
(5)- - p (8)) + -r
Monge (s, - 1 (s,a,r))))))
PT
(137)
log (3) + log (218P144)
S 2/12187 [||R]
(138)
MD mc
K KE my dn
Part
From Lemma
10 4A (0, 1], with probability - 8, denote
max +ES,
PM (-1s, price (-|s, a)ll
(139)
< |S| ||: Keeeslloo
(140)
log (?)
< 2 2|
(141)
Kerainn
čn = max
R (r(sss)) - R ar)
(142)
.ES, REE
< K5,- - Kitestll
(143)
log (2) Kerain
2
(144)
then from Lemma have
H (1l (s) P Kiccet (s))1 + (1,
P puan (s,
(145)
log (+) [Craiin]
4H² ||||||||R
(146)
Overall Combining Part I and Part II, we have
Seaat.De De.e (IT Pout
(147)
MD (s) 13: (5)) p=>(s)) pa
MD wns (s,0,7) f:5 (s,a,r))l)+
H Px
(148)
H (lle (8)
P Keart
(s)ll + 11 (8,a,)) Meant (8,a,r))1
(149)
27

---

Under review as a conference paper at ICLR 2023
log (+) + log (2)5)((A)
log (1) [Kerain|
$ <2H² 1 |A| [R]
+ +2
(150)
dg MDage inte
K Kestape ste
A.6.1 DETAILED LEMMAS (PART I)
Lemma 6. In an MDP M with an offline dataset for any batch- policy π E III,, if | PM (1)s, S cp,
max
(151)
*ES,aEA dr (x,a)0
max a, with pt ADD (s,a)20
| (rls,)) RM (rls, a)) sen
(152)
we have
| (8) Pā (5) + 1100 11,00(s,000 (8, Px M (8, (8,
(153)
s |8| (IA) |R| 1) H - 1
cp |S| Rn
(154)
Proof. For each $ E S, A, create auxiliary reward function F(s,a) : : A - [0. 1]: Vs E E,
1
H' if 8 and à,
i(s,a) a)
(155)
0,
otherwise.
Denote M = A,
R, H. PMP HM, +) and Mp (S. (S,AAR,, H, PMp, f). Using the offline Simulation 2 for any batch-constrained policy E 111",
Lemma shown in Lemma
Px M (8, = (x) and p (8, a) I (T).
(156)
Thus, 6 = 0, Imaz
# and
10a (8, û) - P/
Mo (8, (s.a)] S (T) Int (T))
(157)
H(H 1) 1)rman
< He,
(158)
2
H 1
Epp
(159)
Similary, we have Vs S.
H 1
M (s)
Px
MD (s))
<
2 EP.
(160)
For any s E E, E, E
p (s, - Pr (s,,,r)
(161)
M (s (rls,a) Pr MD (1)
(162)
p MM(s, (s, 0r (s,a) RM (r(s,a) p (s,a) R (rls,a) RD (rls,a))
(163)
S p (s, - PT
- Ma (s, + RM - RMo
(164)
1-1
Ep + CR.
(165)
2
Therefore,
|p M (s) - Mp (s)],, + (s,
- Px MD (8, a r))li
(166)
H 1
< |S]
2 Cp + |S| |A| |R|
2 Cp + CR
(167)
1 1 1
-|8((A4) [ +1)
cp
+
|S| |A|
|R| CR-
(168)
n
28

---

Under review as a conference paper at ICLR 2023
A.6.2 DETAILED LEMMAS (PART II)
Lemma 7 (Simulation Lemma for Finite-Horizon MDPs). Given pair of finite-hrizzon MDPs M1 and M2 with the same state space S, same action space A, same reward function R, and same horizon H. If
max aES,aEA
||
(-1s.0)- - ---pppaaa (M), S Ep,
(169)
max aES,aEA
Ai (s, - ( s tr.
(170)
max sEB,aEA
max (8) (s, a)) S Imaar
(171)
where Mi (s, Eg (s.a) [7] y Nd (ia we have
JM) S Hē, + HHH 1)rmaz
Ep.
(172)
Proof. Similar with famous Simulation Lemma in mnite-horizon MDPs (Alekh Agarwal 2017) and the offline variant shown in Lemma 2
we will prove 4h E [H), Vs E Sn.
V-M (s) VaMa (s) (H h)2, + (H - [) - h - 1)r Imaa čp
(173)
2
by induction. When h H- 1, have Vs E Sn, IVM (s) V/M2(s) & holds. And
wh E [H - 1]. Vs E Sh.
(s) - (s)]
(174)
< Σ T[a|s) (s,
(s, a) - (s,
(175)
AEA
Σ T(als)
Σ
|M aV pM
(176)
oEA
8'ESnt1
(H - h)ll1 - h 1)r max Ep.
< (H h)er +
(177)
Thus,
IJM (T) (T) = -VaMi (5o) -V,Ma (so)
(178)
HHē + H(H 1) 1))max Ep.
(179)
Lemma 8. Given a pair of finite-hhrriie MDPs M1 and M2 with the same state space, same action space, same reward function, and same horizon. If
max
[.M.( - PMa (-Ms, Sep,
(180)
sES,eEA
max EEA
RR (r(s,))))) - R- (r(,, S En:
(181)
we have
(I p (s)(1) + +111 11l1 (s,a,r) - p Ma (s,
(182)
< |S| (|AI|R) + 1)
2 Ep + R
(183)
Proof. Similar with Lemma
Vs E S, E A.r R. using Simulation Lemma
we have
H 1
H- 1
1 (s) - p Ma (s)
2 Ep and
| - p (s, a)) <
2 Ep.
(184)
and
E H-1
1 (s, a, - -p-pp (s.
Ep + ZR.
(185)
2
29

---

Under review as a conference paper at ICLR 2023
Therefore,
1| (8) Px (s) + +110 - (|A]]|)]]
(8, -
2 (s, as,
(186)
+
2 Ep |S| R||ER
(187)
Lemma 9. Let X,Y Y be two i.i.d. random vectors that take values in [0, 1]", N+, For any e E (0, 1). we have
9 2
P
max iE|n
X, - Yi
5
(188)
Proof. Denote an auxiliary set
max
[2:|
<
(189)
iE|n]
then if X, Y V, must have
max X; - Yil <c iE/n]
(190)
For any c E N", denote
where uf = a -(at2) Vie [n].
= V ee,
(191)
We may construct a of such cosets of as follows:
S ( C}2 where C C= c= &
(192)
There are several properties related to these constructions:
For any E N if X, VS, maxie[n] [X, Yil <E
The union of sets in contains [0, 1)m
Any two different sets in S are disjoint.
The only loophole is that have not considered points in boundaries 0Ve (V E S). These boundaries can be decomposed into disjoint union of hyperplanes in R". For each one of the hyperplanes, arbitrarily designate it to adjacent v S. New Vos are the union of the original one and the hyperplanes designated to it. Note that
Σ X EXEV'S =1
(193)
Therefore,
P max Xx - Y3| iE(n)
51- Σ PIXEVVPIIVETs EV9
(194)
CEC
=1 EPX
(195)
cEC
< 1
1
Σ E E
(196)
C
NEEC
(197)
C
Since > 1, we have
1
2
|C|
1
)
5
and
P
max :E|m]
X; V.2
5
5
(198)
30

---

Under review as a conference paper at ICLR 2023
Lemma 10. In transformed BAMDP M* with an offline malti-task dataset D'' .frr any mela-testing task Feest ~ p(K), vg E (0, 1]. with probability - 6, we have
||r. Retestloo
(199)
max (I)Ppn) (s,a,5) 5') pmecot (s,,,,5)1 11 (s,a, Rusor (s, r))l))
(200)
log (t)
((7-1181))))
2 [Kerrinl
(201)
where Ri-- E Ktrain is the closest offline meta-training task to Fiest (see Eq. (25 J. || - Reest llac the distance between Riz- and Ktest (see Eq. (12 and Ktrain is the i.d. offline meta-training tasks sampled from p(k) in D
Proof. From Lemma
we set [S| (1S) + [R]).
then Vc E (0,1), Vig E Kirain.
P [[] - Krest 2d S (
(202)
Hence
P
arg min || - Keeatloo 2
P Illk Keesrle 2
(203)
K.EKivata
( ny [KErrsn
1
(204)
Therefore, v5 E (0, 1], with probability
log (t) Keraanl
arg ||ks, - Kiess llac 52
(205)
Kr, EXttainn
31

---

Under review as a conference paper at ICLR 2023
B HYPER-PARAMETER SETTINGS
Environment Settings. Table shows hyper-parameter settings for the task sets used our exper- iments. Most hyper-paramettrs are adopted from previous works (Li et 20200 Mitchell al.) 2021). For all task sets, 80% of the tasks are meta-training tasks, and the remaining 20% tasks are meta-testing tasks.
Table
Environment parameter settings.
of Adaptation Episodes
# of Trajectories per Task
Environment
Episode Length
# of Tasks
All Meta- World Envs
500
10
50
45
Cheetah- Vell
200
10
100
Point-Rohot
20
20
100
45
Point-Rohot-Spare
20
20
100
45
GCC hyper-parameter settings. Table 4 shows GCC's hyper-paramete settings. Most hyper- parameters are adopted from FOCAL (Li 2020). We set ne to as the evaluation environments all nearly deterministic.
Table 4: Detailed hyper-parameter settings for GCC.
Hyper-Parameter
Hyper-Parameter Values
batch size
256
meta batch size
16
learning rates for dual critic learning rates for all other components
le-4
3e-4
network structure for all components
three fully connected layers with 200 units adam
optimizer
discount
0.99
latent size
20
reward scale
100 for point envs, 1 for all other envs 1/2 of total adaptation episodes
no
1/2 of total adaptation episodes
fhe
32

---

Under review as a conference paper at ICLR 2023
C ANALYSIS AND VISUALIZATION ON UNCERTAINTY ESTIMATION
To further investigate why the uncertainty estimation method (Kidambi et al. 2020) fails to identify in- distribution trajectories, we demonstrate the ensemble's s prediction errors and uncertainty estimation on the first 10 trajectories. As shown in Figure 5 uncertainty estimation methods cannot accurately estimate the distance to the offline dataset, and fail identify in-distribution data. On the other hand, GCC can successfully select in in-distribution data with its greedy selection mechanism.
Epitode Return
Episode Retum
Episcde Return 14.44
Coloode Retum 16.69
13
08
68
-11
Ep:odd 6, Retur 9.12
Epissde 7, Retum Ret7m-19..17 19.17
Eptose 0. Retum -1.44
Episdde 9, Return 7.90
Epeocde 10, Setum 15.47
33-
68-
-25
-10
-33
05
10
10
80
18
-35
-1
65
-05
-10
60
i8
-45
00
(a) GCC
Estiode Unceraaitty
Uncertaiit
Estidde uncertaitty
2.2
33
33
-1.0
Estoode Uncertaiity 0,362
Epbode 7. Uncertliity 0.356
Epiodde Uncerality 0,383
Estode Uncersliity 0.369
Eabode 10, Uncerbanny 0.344
22
-0.5
-55
18
05
10
-1
33
10
15
055
-05
Uncertainty Estimation
Goal a mela-training task
Corresponding oflline dataset
Unselected Iraj
Selected Iraj
Figure 5: Visualization of GCC and uncertainty estimation method's 's trajectory selection. (a) GCC successfully selects the in-distribution trajectory. (b) Uncertainty estimation method cannot identify in-distribution data, as its uncertainty estimation is not accurate.
D ADDITIONAL VISUALIZATION RESULTS
Figure GCC, FOCAL, and uncertainty estimation method (Kidambi et 2020) 's adaptation visualization (episode 1-20) after the initial exploration phase adaptation (episode 1-10). Results demonstrate that while GCC is able to identify in-distribution data and achieve superior adaptation performance, FOCAL utilizes all the 10 trajectories for adaptation, and cannot correctly update task belief due the out-of-distribution issue. The uncertainty estimation method, discussed in Appendix C fails to correctly select the in-distribution trajectory, and thus cannot successfully reach the goal.
33

---

Under review as a conference paper at ICLR 2023
Episede 11, Neturn 53.8
Eibodi 12, Return 5.40
Egrsode 13, Netamn -5.37
Epioodo 14, Neture -5.16
Episode 15, Nolum 55.44
10
60
Lo
66 65
15
10
68
as
10 63 66 16
05
1.5
1.0
-10
(a)
GCC
Eptoode 11,
Episode 12, Retum
Episade 13,
Eplecde M. Ratum -7.92
Eptodd
1.5
15
15
16-
15-
-10
08
-55
-12
100
10-
10-
10-
05-
15
65
15
65
(b) FOCAL
Pobaade 11,
Epieade 12, Ratum
Falende
Poteade 14, 28.76
Ppiunde
1.5
13
10-
<Ls 68
La
-12 6e
as
1.0
10
13
10
05
65
05
(c) Uncertainty Estimation
Goal for meta-training task
Corresponding offline dataset
First 10 episodes' traj (filtered out) First 10 episodes' traj (selected)
Traj for current episode
Figure 6: Visualization of GCC, FOCAL and uncertainty estimation method's adaptation in Episode 11-20 on the Point-Robot environment. (a) GCC successfully selects the in-distribution trajectory and keeps improving in adaptation. (b) FOCAL suffers from the distribbutt issue, and cannot correctly update posterior belief, leading to poor performance. (c) Uncertainty estimation method cannot identify in-distribution data, and also suffers from the out-of-distribution issue.
34

---

Under review as a conference paper at ICLR 2023
E ADDITIONAL EXPERIMENT RESULTS
E.1 UNCERTAINTY ESTIMATION METHOD ON REPRESENTATIVE TASKS
Table shows the uncertainty estimation method's online adaptation performance on several represen- tative tasks. Results demonstrate that the uncertainty estimation method achieves similar performance FOCAL, and underperforms GCC. This because the uncertainty estimation method cannot
correctly estimate in-distribution trajectories, discussed in Section 5.1 Appendix C Candd D
Table Comparison between GCC, FOCAL, and the uncertainty estimation method's performance on several representative tasks. For Meta-Worke World tasks we show normalized scores.
Env
GCC
Uncertainty Estimation
FOCAL
Plate-Slide-Back
0.96 + 0.02
0.64 + 0.21
0.58 + 0.06
Hammer
0.84 + 0.06
0.61 + 0.33
0.59 + 0.07
Coffee-Push
1.26 + 0.13
0.02 = 0.01
0.66 = 0.07
Push-Wall
0.71 = 0.15
0.48 = 0.19
0.43 = 0.06
Point-Robot
-5.10 + 0.26
-19.62 + 0.77
-15.38 + 0.95
E.2 DETAILED ALGORITHM PERFORMANCE ON ALL TASKS
Table Table 7 and Table g shows baselines' online adaptation and offline performance on all 50 Meta- World MLL task sets, respectively. GCC significantly outperforms baselines with online adaptation, and achieves better or comparable performance to baselines with offline adaptation.
Meta-World
35

---

Under review as a conference paper at ICLR 2023
Table 6: Comparison between FOCAL and MACAW with online adaptation and GCC. For Meta-
World tasks. "-V2" is omitted for brevity.
Environment
GCC
FOCAL
MACAW
Coffee-Push
1.26 + 0.13
0.66
0.07
0.01 + 0.01
Faucet-Close
1.12 + 0.01
1.06 + 0.02
0.07 + 0.01
Door-Unlock
1.11 - 0.02
0.97 + 0.03
0.11 + 0.01
Plate-Slide-Side
1.07 - 0.08
0.70 0.14
0.00 1 0.00
Faucet-Open
1.05 - 0.02
1.01 + 0.02
0.08 - 0.04
Button-Presss Wall
1.04 0.04
0.99 0.06
0.02 - 0.00
Plate-Slide
1.01 - 0.03
0.83 + 0.09
0.01 + 0.00 0.00
Door-Close
0.99 + 0.00
0.97 0.01
0.00 0.00
Cower-Closc
0.99 = 0.02
0.96 + 0.04
0 0.50
Plate-Slide- Slide-Back- Bark--ick-ee
0.97 + 0.02
0.77 0.20
0.02 = 0.01
Door-Lock
0.97 + 0.01
0.90 + 0.02
0.25 : 0.11
Window-Open
0.96 + 0.02
0.81 0.07
0.15 + 0.11
Door-Open
0.96 + 0.02
0.78 + 0.13
0.06 + 0.01
Plate-Slide-Back
0.96 - 0.02
0.58 + 0.06
0.21 + 0.17
Window-Close
0.94 + 0.01
0.79 0.01
0.54 + 0.44
Reach-Wall
0.93 = 0.05
0.53 + 0.18
0.82 = 0.02
Dial-Turn
0.91 0.05
0.84 0.09
0.00 - 0.00
Sandle-Press-See
0.91 + 0.02
0.79 + 0.10
0.51 + 0.41
Handle-Pull
0.90 0.02
0.67 0.03
0.00 + 0.00
Handle-Press
0.88 - 0.05
0.87 + 0.02
0.28 + 0.10
Reach
0.85 1 0.03
0.62 0.05
0.63 1 0.04
Lever-Pull
0.85 + 0.02
0.73 + 0.07
0.20 + 0.16
Hammer
0.84 = 0.06
0.59 0.07
0.10 = 0.01
Drawer-Open
0.82 = 0.06
0.64 + 0.10
0.11 - 0.02
Sweep
0.77 + 0.04
0.32 + 0.08
0.20 + 0.20
Button-Press
0.74 + 0.08
0.68 + 0.14
0.02 + 0.01
Stick-Push
0.73 - 0.09
0,46 + 0.15
0.17 + 0.17
Coffee-Button
0.73 + 0.14
0.66 0.16
0.15 - 0.13
Push-Wal
0.71 + 0.15
0.43 + 0,06
0.23 0.18
Shelf-Place
0.70 0.18
0.32 0.11
0.01 + 0.01
Basketball
0.64 - 0.15
0.41 + 0.24
0.00 0.00
Hand-Insert
0.63 + 0.04
0.29 0.07
0.02 0.01
Sweep-Into
0.61 + 0.06
0.33 + 0.05
0.00
Button-Press-Tope Topdown
0.57 = 0.11
0.45 - 0.06
0.38 0.36
Peg- Peg-Unplug-Sice
0.56 + 0.07
0.19 - 0.09
0.00 0.00
Assembly
0.55 + 0.13
0.28 0.05
0.33 1 0.01
Push
0.55 + 0.10
0.34 0.14
0.28 + 0.19
Bin-Picking
0.53 + 0.16
0.31 + 0.21
0.66 + 0.11
Push-Back
0.52 + 0.05
0.16 0.04
0.00 1 0.00
Box-Close
0.51 + 0.11
0.15 1 0.09
0.36 + 0.11
Soccer
0.44 0.04
0.11 0.03
0.38 + 0.31
Button-Press- Prress-TTpdown---e Wall
0.43 - 0.03
0.40 + 0.07
0.05 + 0.02
Disassemble
0.42 0.14
0.26 0.04
0.05 + 0.00
Coffee-Pull
0.40 + 0.05
0.23 + 0.04
0.19 0.12
Stick-Pull
0.32 0.06
0.17 0.07
0.00 + 0.00
Peg-Insert-Side
0.30 + 0.04
0.08 1 0.03
0.00 0.00 0.00
Pick-Place- Wall
0.28 + 0.12
0.09 0.04
0.39 1 0.25
Pick-Out-Of-Holl
0.26 - 0.25
0.16 + 0.16
0.59 + 0.06
Pick-Place
0.20 - 0.03
0.07 0.02
0.05 - 0.05
Handle-Pull-Side
0.14 + 0.04
0.13 0.09
0.00 + 0.00
Average
0.73
0.07
0.53
0.08
0.18 + 0.09
Cheetah- Vel
-17..52 = 21.96
-287.70
+ 30.62
-23,977
23.46
Point-Robot
-5.10 + 0.26
-15.38 + 0.95
-14.61
+ 0.98
Sott-Robot-Sparee
7.78 + 0.64
0.83
0.37
0.00 + 0.00
36

---

Under review as a conference paper at ICLR 2023
Table 7: Comparison between BOReL and GCC, For Meta-Worid tasks, "-V2" is omitted for brevity.
Environment
GCC
BOReL
Coffee-Push
1.26 + 0.13
0.00 = 0.00
Faucet-Close
1.12 + 0.01
0.13 0.03
Door-Unlock
1.11 + 0.02
0.13 = 0.03
Sate-Slide-Sitee
1.07 0.08
0.00 0.00
Faucet-Open
1.05 + 0.02
0.12 0.05
Button-Press Prent-Prsss--ie Wall
1.04 0.04
0.01 0.00
Plate-Slide
1.01 + 0.03
0.01 = 0.00
Door-Close
0.99 + 0.00
0.37 = 0.19
Drawer-Close
0.99 + 0.02
0.00 0.00
Saa--Siid-BBack---e
0.97 + 0.02
0.01 = 0.00
Door-Lock
0.97 + 0.01
0.14 0.00
Window-Open
0.96 + 0.02
0.03 = 0.00
Door-Open
0.96 + 0.02
0.12 0.01
Plate-Slide-Back
0.96 + 0.02
0.01 = 0.00
Window-Clow
0.94 0.01
0.03 0.00
Reach-Wall
0.93 - 0.05
0.06 0.00
Dial-Turn
0.91 + 0.05
0.00 = 0.00
Handle- Handle-Press-ee Press-Side
0.91 + 0.02
0.02 0.01
Handle-Pull
0.90 0.02
0.00 0.00
Handle-Press
0.88 + 0.05
0.01 0.00
Reach
0.85 1 0.03
0.04 = 0.01
Lever-Pull
0.85 + 0.02
0.05 0.00
Hammer
0.84 + 0.06
0.09 = 0.01
Drawer-Open
0.82 + 0.06
0.10 0.00
Sweep
0.77 + 0.04
0.00 = 0.00
Button-Press
0.74 + 0.08
0.01 0.01
Stick-Push
0.73 + 0.09
0.00 = 0.00
Coffee-Button
0.73 0.14
0.02 0.00
Push Wall
0.71 - 0.15
0.00 = 0.00
Shelf-Place
0.70 0.18
0.00 = 0.00
Basketball
0.64 + 0.15
0.00 = 0.00
Hand-Insert
0.63 0.04
0.00 0.00
Sweep-Into
0.61 + 0.06
0.01 0.00
Button-Press- Putton-Press-Toodoww
0.57 + 0.11
0.02 = 0.02
Peg-Unplug-Sidee
0.56 + 0.07
0.00 0.00
Assembly
0.55 + 0.13
0.04 = 0.00
Push
0.55 + 0.10
0.00 0.00
Bin-Picking
0.53 + 0.16
0.00 = 0.00
Push-Back
0.52 + 0.05
0.00 0.00
Box-Close
0.51 + 0.11
0.05 0.01
Soccer
0.44 0.04
0.04 = 0.02
Button-Press- -oteeess--Tooooo--WWaaa
0.43 + 0.03
0.05 = 0.01
Disassemble
0.42 0.14
0.04 0.00
Coffee-Pull
0.40 + 0.05
0.00 = 0.00
Stick-Pull
0.32 1 0.06
0.00 = 0.00
Peg-Insert-Side
0.30 + 0.04
0.00 0.00
Pick-Place- Wall
0.28 - 0.12
0.00 = 0.00
Pick-Out-Of-Hoo
0.26 - 0.25
0.00 0.00
Pick-Place
0.20 + 0.03
0.00 = 0.00
Handle-Pull-Side
0.14 + 0.04
0.00 0.00
Average
0.73 0.07
0.04
0.01
Cheetah-
-171.52 + 21.96
-301.4 + 36.8
Point-Rohs
-5.10 + 0.26
-17.28 + 1.16
SointRRobot-Sparee
7.78 + 0.64
0.00 = 0.00
37

---

Under review as a conference paper at ICLR 2023
Table 8: Comparison between baselines with offline adaptation and GCC. For Meta-World tasks,
V-V2" is omitted for brevity.
FOCAL with Expert Context
MACAW with Expert Context
Environment
GCC
Coffee-Push
1.26 + 0.13
0.50 + 0.06
1.14 + 0.27
Faucet-Close
1.12 - 0.01
1.07 + 0.02
1.01 +-2 0.01
Door-Unlock
1.11 - 0.02
0.96 0.03
0.99 + 0.04
Wate-Slide-Side
1.07 - 0.08
0.75 + 0.09
0.91 + 0.09
Faucet-Open
1.05 0.02
1.00 0.02
0.99 = 0.01
Button-Pres--Wal Wall
1.04 + 0.04
0.98 + 0.05
0.99 + 0.01
Plate-Slide
1.01 + 0.03
1.00 0.03
0.67 + 0.07
Door- -lo--Close
0.99 = 0.00
0.97 + 0.01
0.92 - 0.05
Drawer-Close
0.99 - 0.02
0.96 0.04
1.00 = 0.01
Plate-Slide-Bade -iddee
0.97 = 0.02
0.90 + 0.07
0.80 + 0.05
Door-Lock
0.97 + 0.01
0.88 0.04
0.97 + 0.03
Window-Ope
0.96 + 0.02
0.93 + 0.05
0.98 + 0.02
Door-Open
0.96 + 0.02
0.90 + 0.02
0.99 + 0.02
Plate-Slide-Bace
0.96 + 0.02
0.93 0.01
0.55 1 0.11
Window-Close
0.94 - 0.01
0.73 + 0.02
1.00 = 0.01
Reach-Wall
0.93 0.05
0.91 0.05
0.82 - 0.02
Dial-Turn
0.91 + 0.05
0.84 + 0.08
0.98 + 0.01
Handle-Press Press---eess
0.91 0.02
0.87 0.04
0.82 + 0.10
Handle-Pull
0.90 - 0.02
0.70 + 0.05
0.95 + 0.05
Handle-Press
0.88 1 0.05
0.79 0.08
0.56 + 0.19
Reach
0.85 + 0.03
0.83 1 0.05
0.64 + 0.08
Lever-Pull
0.85 = 0.02
0.76 0.03
0.97 = 0.07
Hammer
0.84 = 0.06
0.78 + 0.04
0.40 = 0.18
Drawer-Open
0.82 + 0.06
0.73 + 0.11
0.98 + 0.01
Sweep
0.77 + 0.04
0.74 + 0.02
0.98 + 0.01
Button-Press
0.74 - 0.08
0.63 + 0.09
0.71 - 0.04
Stick-Push
0.73 + 0.09
0.14 0.09
0.67 1 0.09
Coffec-Button
0.73 + 0.14
0.61 + 0.20
0.21 - 0.11
Push-Was hall
0.71 0.15
0.90 0.12
0.96 + 0.09
Shelf-Place
0.70 - 0.18
0.57 + 0.13
0.55 - 0.03
Basketball
0.64 + 0.15
0.49 0.17
0.47 4 0.18
Hand-Insert
0.63 + 0.04
0.64 + 0.09
0.20 + 0.09
Sweep-Into
0.61 = 0.06
0.64 0.09
0.00 + 0.00
0.57 + 0.11
0.48 + 0.11
0.92 - 0.04
Peg-Unplug-Sice
0.56 + 0.07
0.57 0.10
0.18 + 0.10
Assembly
0.55 + 0.13
0.64 + 0.03
0.36 + 0.02
Push
0.55 + 0.10
0.98 + 0.13
0.86 + 0.02
Bin-Picking
0.53 + 0.16
0.61 0.12
0.63 1 0.11
Push-Back
0.52 + 0.05
0.52 1 0.16
0.15 + 0.09
Box-Close
0.51 0.11
0.56 0.08
0.35 - 0.11
Soccer
0.44 + 0.04
0.45 + 0.03
0.59 - 0.11
Prees-TTopdow--Ve Wall
0.43 0.03
0.40 0.06
0.43 + 0.06
Disassemble
0.42 + 0.14
0.23 + 0.05
0.50 + 0.06
Coffee-Pull
0.40 0.05
0.58 0.11
0.45 + 0.11
Stick-Pull
0.32 + 0.06
0.18 1 0.06
0.27 1 0.09
Peg-Insert-Side
0.30 + 0.04
0.52 0.08
0.25 1 0.04
Pick-Place- Wall
0.28 + 0.12
0.13 - 0.07
0.21 --40000 0.16
Pick-Out-Of-Holl
0.26 = 0.25
0.27 - 0.27
0.59 - 0.08
Pick-Place
0.20 + 0.03
0.29 0.11
0.72 + 0.09
Pandle-Pull-Side
0.14 + 0.04
0.09 + 0.05
0.94 + 0.08
Average
0.73 + 0.07
0.67 + 0.07
0.68 + 0.07
Cheetah- Vel
-171.52 + 21.96
-156.07 + 23.22
292.992 + 36.66
Point-Robot
-5.10
+ 0.26
-4.68
+ 0.18
-19.60 1.15
Point-Ron Sont-Rooot-Sparre
7.78 + 0.64
8.37
0.67
0.00 + 0.00
38

---

Under review as a conference paper at ICLR 2023
F ABLATION STUDY
In this section, we conduct various ablation studies to investigate the robustness of GCC in dataset quality and typer-parameters.
GCC's performance with different initial stage lengths. The total number of adaptation episodes 20. We find that GCC performs well during 10-15 episodes, which is 50%-75% of the total number of adaptation episodes. A small initial stage length (5) may lead to possibly unreliable task belief and cause a degrade performance. The 19-episode does not perform the iterative optimization process, and the task belief updates will not converge.
Initial stage length.
Number of latent task variables sampled in the initial phase. n; controls the number of diverse samples used to produce the task embedding candidates 2/. As shown in Table 10 GCC is robust to changes of ng, and works in large range from 20.
Dataset Quality. We test GCC and baselines with several "medium" datasets, which are collected by periodically evaluating policies of SAC. As shown in Table TT] GCC still significantly outperforms baseline algorithms on medium datasets, which implies GCC's ability to learn various datasets. We do not test BOReL, as already fails on the casier expert-level datasets.
Table 9; GCC's performance with various initial stage lengths.
Environment -6.04 + 0.31
Episodes
10 Episodes
15 Episodes
19 Episodes
Point-Roof Roott
-5.11
+ 0.21
-5.10
+ 0.26
-5.37
+ 0.11
Doint-Robot-Sparse
4.04 + 0.58
7.78
= 0.64
8.07
0.62
7.29
= 0.50
Table 10: GCC's performance with various n2s8.
Environment
7g
71g 5
Y1g
10
71g 20
Point-Robot
-5.92 + 0.31
-5.11
0.21
-4.94
+ 0.16
-4.99
0.23
Point-Robot-Sppe
5.66 = 0.63
7.78 + 0.64
7.31
+ 0.74
7.78
+-2 0.57
Table 11: Algorithms' performance on datasets of various qualities.
Environment
GCC
FOCAL
MACAW
Sweep
0.77 0.04
0.32
0.08
0.20 = 0.20
Sweep-Medium
0.59 = 0.13
0.38 + 0.13
0.04 = 0.03
Peg-Insert-Side
0.30 = 0.04
0.08 + 0.03
0.00 = 0.00
Peg-Innert-Side-MMddin
0.30 + 0.14
0.10 + 0.07
0.00 0.00
39


Reviews:
[{"review_id": "ayr-kyV41Y", "reviewer": "Reviewer_7zMW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work considers the problem in offline meta-reinforcement learning, where there is a distribution shift between the contexts seen in the offline data, and the contexts obtained by online roll-outs. This work first offers some formal characterizations of this shift, and then uses these characterizations to motivate a new offline meta-RL algorithm that works by filtering out episodes that appear out-of-distribution in order to obtain contexts that still look in-distribution for online adaptation.", "summary_of_the_review": "Overall, the ideas in this work are quite interesting and are useful to the community. However, as there are some significant weaknesses to address, I will begin by only weakly recommending acceptance, though I am willing to increase my score based on the authors' response.", "strength_and_weaknesses": "## Strengths\n\nThere is much to like about this work. This work clearly and soundly characterizes the problem that it is trying to solve (with some exceptions, described below), and overall, the main idea of filtering out trajectories that the policy doesn't know how to update on from online contexts is a novel and interesting approach to offline meta-RL. Additionally, this approach appears to yield fairly impressive empirical improvements that are relevant to the meta-RL community.\n\n## Weaknesses\n\nThe theory could be strengthened in a few areas:\n- Specifically, the notion of filtering introduced in Fact 1 is insufficiently precise and requires some work to formalize. One precise notion of filtering is to filter based on visiting belief states under the transformed BAMDP that are also visited by the behavior policies (i.e., are present in the offline data). However, this still requires some work to formalize: how do you deal with multi-step episodes, where the first few timesteps include belief states that are out-of-distribution from the offline data, but later transition into in-distribution beliefs? Intuitively, inclusion of such episodes seems beneficial for the post-adaptation policy, but it requires more precise filtering definitions.\n- It should also be noted that the existing notion of filtering based on visiting transformed belief states is generally infeasible. For example, in the case where different behavior policies take distinct actions at the initial state, but the observation after taking these actions does not reveal information about the identity of the current MDP, the transformed belief update will erroneously place all mass on the MDP corresponding to the observed behavior policy, which cannot be corrected without actually knowing the identity of the MDP. At the very least, it would be worthwhile to discuss under what cases such filtering is actually possible.\n\nThis work would also benefit from discussion about its own limitations and acknowledgements of which aspects are heuristics or approximations. Specifically, the idea to practically filter in GCC based on the trajectories earning the highest reward is heuristic that does not work in general. In many tasks requiring \"challenging\" exploration, such as those considered in HyperX [1] or DREAM [2], highly informative exploration trajectories do not necessarily achieve high returns, and would be filtered away in GCC. Other times, it may be impossible or difficult to obtain any high reward trajectories, even when these trajectories are highly informative, which would also cause failure. This work would be strengthened from discussion on this aspect. Additionally, it may be interesting to discuss relationships with the original theoretical motivation of matching behavior policy trajectories or more precisely, testing for in-distribution-ness with the behavior policies.\n\n[1] Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning. Zintgraf et al., '21.\n\n[2] Decoupling Exploration and Exploitation in Meta-Reinforcement Learning without Sacrifices. Liu et al' 21.\n\n## Additional Comments\n- There is an error in Proposition 1, where the policy evaluation gap is in fact $\\frac{H^+ - 1}{2}$, and not $\\frac{H^+}{2}$.\n- It's unclear why the example in Figure 2 includes $2v$ actions, where the last $v$ actions yield no reward -- this appears to have no bearing whatsoever to the provided theory, and could probably be omitted for clarity.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OCpjByhBmt", "reviewer": "Reviewer_HYwC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a greedy context-based meta-testing procedure to tackle the distribution shift issue when performing online adaptation of offline meta-trained policies. The proposed method extends FOCAL with a diverse latent task embedding sampling strategy and a context selection mechanism based on the highest return in the meta-testing phase. The authors justify such a methods with analysis in the BAMDP framework. The experiment results in several standard offline meta RL benchmark datasets demonstrate the superiority of the proposed method over previous offline meta-RL methods in the online adaptation setting. ", "summary_of_the_review": "This is a borderline paper and I’m willing to adjust my score if my concerns outlined in my above comments (mostly regarding clarity) could be addressed. ", "strength_and_weaknesses": "### Strength\n\n1. The proposed method is well-justified by the theoretical analysis and the proposed method performs well compared to baseline methods. \n\n### Weaknesses\n\n1. The clarity needs to be improved (see my detailed comments below).\n2. There’s no discussion on the limitations of the proposed method. Specifically, the proposed method still assumes that the meta-testing task is sampled from the same task distribution as the meta-training tasks. Thus, the method is dealing with data distribution shift between offline data and online adaptation data, but not distribution shift between meta-training tasks and the meta-testing task (e.g., in cheetal-vel when all meta-training tasks correspond to velocity targets < 1 m/s while the meta-testing target velocity is 2 m/s). I encourage the authors to make it more clear in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5wbdW50ygg", "reviewer": "Reviewer_VSon", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper identifies the distribution shift between the offline dataset and online rollouts as the core problem for offline meta-RL and adopts a Bayesian inference procedure to handle it.", "summary_of_the_review": "The paper proposes a data distribution correction method to efficiently 'guess' a probable task during policy deployment. While the clarity of the paper can be improved, the proposed method is intuitive and shows better empirical performance compared to the baselines.", "strength_and_weaknesses": "Strength\n- Empirical performance is strong compared to the baselines.\n- The proposed method is intuitive.\n\nWeakness\n- The authors introduces the term \"data distribution mismatch\" and describes it as a challenge unique to offline meta-RL. However, I am confused with this term in two ways: 1) \"data distribution mismatch\" seems too broad a term to indicate a problem specific for offline meta-RL. 2) It is unclear why this problem is unique to offline meta-RL. Distribution shift is a problem that persists for almost any learning models. In what aspect is data distribution match different from distribution shift? In other words, why do we need this new term?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3nHyQPRmqH", "reviewer": "Reviewer_NxHR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles offline meta-RL where new tasks are learned via online few-shot adaptation. They observe that there is a distribution shift between the offline data used for meta-training, and the data collected online by the meta-learned policy to use for adaptation. This distribution shift can result in poor adaptation performance. To remedy this, the paper proposes to adapt the policy using only the trajectories with the highest returns - using return as a proxy for out-of-distribution detection. The method is evaluated on the MetaWorld Benchmark where it outperforms prior approaches. \n\nContributions\nan offline meta-RL algorithm that can adapt online to new tasks within a few trajectories, without knowledge of the reward function or offline data for the task.\nDefine the distribution shift between behavior policy and learned policy and show that OOD context data leads to poor adaptation\nEmpirical evaluation demonstrating that the proposed algorithm outperforms prior approaches on a standard meta-RL benchmark", "summary_of_the_review": "The paper tackles a relevant and important problem in the field of meta-RL -- addressing the distribution shift present when meta-training offline and learning new tasks online. The experimental results show significant improvements over prior work. However, my current understanding of the method is that it is not able to distinguish between OOD trajectories and trajectories that achieve low return in a given test task, and thus can only adapt from high-return trajectories, severely limiting the applicability of the method. Therefore I don't think it currently meets the bar for a contribution for this conference.\n\n--------- Update 12/6/22 ----------\n\nThank you to the authors for your detailed response, which has clarified a misunderstanding I had about the method. There is an assumption that the offline meta-training dataset is collected by expert policies, and thus only contains high-return trajectories. In my first review, I missed this assumption, which is stated in Section 2.2. I would recommend that the authors make this assumption more explicit in other parts of the paper as well. For example, the introduction refers to “task-dependent” policies, but it is not clear that this implies expert policies. \n\nWith this clarification, I understand that part of the distribution shift being addressed is between high-return trajectories during meta-training and potentially low-return trajectories collected during meta-testing. Figure 3 is now clear, thank you!\n\nHowever, the proposed approach of handling this distribution shift by filtering out low-return trajectories has great limitations that in my view diminish the contribution significantly. The method cannot make use of low-return exploration trajectories to update the task belief, so exploration during meta-testing is constrained to sampling the unadapted meta-learned policy. Thus the method effectively does not address the problem of exploration in meta-RL, which is a key challenge. While the experimental results do show improvement over prior methods (the prior methods are not designed for online adaptation and so are severely handicapped, but I am not aware of an existing work that is), I think the methodological limitations are severe enough to outweigh the empirical contribution. \n\nI think clarity of presentation can be further improved; specifically, to make the problem statement very clear. I also find the notation in the theory presentation to be difficult to follow. Limitations of the method should be clearly discussed in the main body of the paper. \n\nOverall, I think that the methodological limitations outweigh the contributions, and that the paper would make a much stronger contribution if this limitation is addressed. Therefore, I maintain my score. ", "strength_and_weaknesses": "Questions / Comments\n1) If I understand correctly, the proposed method is to use the returns of few-shot trajectories collected at test time as a proxy for measuring whether the collected data is in distribution. Then adaptation is performed using only the highest-return trajectories as context. In Section 4.2, the motivation for this proxy is stated in one sentence, referencing prior works Fujimoto et al. 2019 and Kostrikov et al. 2021. However, this proxy seems like quite a blunt instrument, conflating areas of the state-action space with low return in a particular task with areas outside the support of the offline data distribution. Consider the didactic example in Section 5.1 - the agent cannot make use of trajectories that don’t reach the goal to update its task belief, because those trajectories are deemed OOD because they have low return? I think I am mis-understanding the method, could the authors please clarify? If I am understanding correctly, then the method is quite limited. \n\n2) It would strengthen the paper to show a baseline that tries to detect OOD data in a more standard fashion, e.g., estimating uncertainty via an ensemble.\n\n3) In the point robot experiment in Figure 3, if the training data is all clustered around a trajectory that goes straight (light blue bubbles), how is it that the exploratory trajectories sampled online navigate coherently outside of this distribution (green dots)? As an aesthetic nitpick, 3a is quite small and hard to parse.\nNitpick in wording: On page 1, “BORel (Dorfman et al., 2021) and SMAC (Pong et al., 2022) employ few-shot online adaptation, but the former assumes known reward functions, and the latter requires free interactions with the environment without reward supervision.” It seems that the advantage of the proposed method over SMAC is that thousands of samples (albeit unsupervised) from the test environment are not required. This sentence doesn’t really say that, and makes it sound as if collecting data without reward supervision is onerous, which is not the case. \n\n4) In Section 6, how is the distribution shift discussed in this paper different from the distribution shift discussed in Pong et al. and Dorfman et al? I’m not as familiar with Dorfman et al, but in the case of Pong et al. it seems that this paper is tackling exactly the same issue as raised in that paper. It’s not a problem to tackle the same problem as other papers, and can lead to confusion not state so clearly.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject