FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization
Authors: Lanqing Li
Rui Yang
Dijun Luo
Venue: ICLR 2021 Poster (2021)
Keywords: offline/batch reinforcement learning
meta-reinforcement learning
multi-task reinforcement learning
distance metric learning
contrastive learning

Abstract: We study the offline meta-reinforcement learning (OMRL) problem, a paradigm which enables reinforcement learning (RL) algorithms to quickly adapt to unseen tasks without any interactions with the environments, making RL truly practical in many real-world applications. This problem is still not fully understood, for which two major challenges need to be addressed. First, offline RL usually suffers from bootstrapping errors of out-of-distribution state-actions which leads to divergence of value functions. Second, meta-RL requires efficient and robust task inference learned jointly with control policy. In this work, we enforce behavior regularization on learned policy as a general approach to offline RL, combined with a deterministic context encoder for efficient task inference. We propose a novel negative-power distance metric on bounded context embedding space, whose gradients propagation is detached from the Bellman backup. We provide analysis and insight showing that some simple design choices can yield substantial improvements over recent approaches involving meta-RL and distance metric learning. To the best of our knowledge, our method is the first model-free and end-to-end OMRL algorithm, which is computationally efficient and demonstrated to outperform prior algorithms on several meta-RL benchmarks.

Published as a conference paper at ICLR 2021
FOCAL: EFFICIENT FULLY-OFFLINE META- REINFORCEMENT LEARNING VIA DISTANCE METRIC LEARNING AND BEHAVIOR REGULARIZATION
Lanqing Li'; Rui Yang"! Dijun Luo-
1 Tencent AT Lab
2 Department of Automation, Tsinghua University
Bangingli1993emmee com, di jijjunlusttencent...
Langruii9@mailee www.gaaaiasssiiiineeeeeeme
ABSTRACT
We study the offline meta-reinforcement learning (OMRL) problem, a paradigm which enables reinforcement learning (RL) algorithms to quickly adapt to unseen tasks without any interactions with the environments, making RI. truly practical in many real-world applications. This problem is still not fully understood, for which two major challenges need to be addressed. First, offline RL usually suffers from bootstrapping errors of out-of-distribution state-actions which leads to divergence of value funetions. Second, me requires efficient and robust task inference learned jointly with control policy. In this work, we enforce behavior regular- ization learned policy
general approach to offline RL, combined with deterministic context encoder for efficient task inference. We propose a novel negative- powerr distance metric bounded context embedding space, whose gra- dients propagation is detached from the Bellman backup. We provide analysis and insight showing that some simple design choices can yield substantial improve- ments over recent approaches involving meta-RL and distance metric learning. To the best of our knowledge, our method is the first model-free and end-to-end OMRL algorithm, which is computationally efficient and demonstrated outper- form prior algorithms on several meta-RL benchmarks
INTRODUCTION
Applications of reinforcement learning (RL) in real-world problems have been proven successful in
many domains such as games (Silver et al. 2017 Vinyals et al.) 2019; Ye et al. 2020) and robot control (Johannink et al. 2019). However, the implementations so far usually rely on interactions with either real simulated environments. In other areas like healtheare (Gottesman et al. 2019),
autonomous driving (Shalev-Shwartz et al. 2016) and controlledeenvirommenn agriculture (Binas) al. 2019) where RI. shows promise conceptually in theory, exploration in real environments is evidently risky, and building high-fidelity simulator can be costly. Therefore a key step towards more practical RL algorithms is the ability to learn from static data. Such paradigm, termed "offline RL" or "batch RL", would enable better generalization by incorporating diverse prior experience. Moreover, by leveraging and reusing previously collected data, off- off-policy algorithms such as SAC (Haarnoja et al. (018) has been shown to achieve far better sample efficiency than on-policy meth- ods. The same applies to offline RL algorithms since they are by nature off-policy.
The aforementioned design principles motivated surge of recent works offline/batch RL Fuuji- moto et al. 2019 Kumar et al.) 2019 Wu al. 2019 (ieeel et al. 2020). These papers propose remedies by regularizing the learner to stay close to the logged transitions of the training datasets, namely the behavior policy, in order to mitigate the effect of bootstrapping error (Kumar et al. 2019), where evaluation errors of out-of distiiibittttinn sate-action pairs are never corrected and hence easily diverge due to inability to collect new data samples for feedback. There exist claims that offline RL.
"Correspondence Lanqing Li dangingli19933 0at Dijun Luo <dijunluo tencent.com>>
Work done while an intern at Tencent Al Lah.
Source code: httptt/////// http://gggiaaabcoooooocc.....
www.:///o...oo. ICUUR/POCAI.-

---

Published as a conference paper at ICLR 2021
can be implemented successfully without explicit correction for distribution mismatch given suffi- ciently large and diverse training data (Agarwal et al. (020). However, we find such assumption un- realistic in many practices, including experiments. In this paper, to tackle the out-of -ut-o--distribution problem in offline RI. in general, we adopt the proposal of hehavior regularization by Wu et al. (2019).
For practical RL, besides the ability to learn without exploration, it's also ideal to have an algo- rithm that can generalize to various scenarios. To solve real-world challenges in multi-task setting, such as treating different diseases, driving under various road conditions or growing diverse crops in autonomous greenhouses, robust agent is expected to quickly transfer and adapt to unseen tasks, especially when the tasks share common structures, Meta-learning methods (Vilalta & Drissi 2002
Thrun & Prati 2012) address this problem by learning an inductive bias from experience collected across a distribution of tasks, which can be naturally extended to the context of reinforcement learn- ing- Under the umbrella of this so-called meta-RL., almost all current methods require on-policy data during either both meta-training and testing phases (Wang et al. 2016 Duan al. 2016 Finn ctt al. 2017) or at least testing stage (Rakelly et al. 2019) for adaptation. An efficient and robust method which incorporates both fully-offline learning and meta-learning in RL, despite few attempts (Li al. 2019b Dorfman & Tamar 2020), has not been fully developed and validated.
In this paper, under the first principle of maximizing practicality of RL algorithm, propose an efficient method that integrates task inference with RL algorithms in a fully-offline fashion. Our fully-offline context-bate based actor-critic meta-RL algorithm, FOCAL, achieves excellent sample efficiency and fast adaptation with limited logged experience, on a range of deterministic continuous control meta-environments. The primary contribution of this work is designing the first end-to-end and model-free offline meta-RL algorithm which is computationally efficient and effective without any prior knowledge of task identity or reward/dynamics. To achieve efficient task inference, we propose an inverse-power loss for effective learning and clustering of task latent variables, in analogy to coulomb potential in electromagnetism, which is also unseen in previous work. We also shed light on the specific design choices customized for OMRL. problem by theoretical and empirical analyses.
2 RELATED WORK
Meta-RL Our work FOCAL builds upon the meta- meta-learning framework in the context of reinforce- ment learning. Among all paradigms of meta-RL, this paper is most related to the context-based and metric-based approaches. Context- based meta-RL. employs models with memory such as recurrent (Duan al., [2016] Wang et al. 2016: Fakoor et al. 2019),, recursive (Mishra et al.] (0117) or proba- bilistic (Rakelly et al. 2019) structures to achieve fast adaptation by aggregating experience into a latent representation on which the policy is conditioned. The design of the context usually leverages
the temporal or Markov properties of RL problems.
Metric-based me focuses on learning effective task representations to facilitate task inference and conditioned control policies, by employing techniques such as distance metric learning (Yang & Jin 2006). Koch et al. (2015) proposed the first metric-based meta-algorithm for few-shot learning. in which a Siamese network (Chopra et al. (2055) is trained with triplet loss to compare the similarity between a query and supports the embedding space. Many metric-based meta-RL algorithms
extend these works (Snell et al. 2017 Sung et al. 2018 Li al. 2019a).
Among all aforementioned meta-learning approaches, this paper is most related to the context- based PEARL algorithm (Rakelly et (019)) and metric- prototypical networks (Snell et al.) 2017]. PEARL achieves SOTA performance for off off-policy meta-RL RL by introducing probabilis- tic eermutation-invariann context encoder, along with design which disentangles task inference and control by different sampling strategies. However, requires exploration during meta-testing. The prototypical networks employ similar design of context encoder as well as an Euclidean dis- tance metric on deterministic embedding space, but tackles meta-learning of classification tasks with squared distance loss as opposed the inverse-power loss in FOCAL for the more complex OMRL problem.
OffchheBBatch RL To address the bootstrapping error (Kumar et al. 2019) problem of offline
RL, this paper adopts behavior regularization directly from Wu et al. (2019). which provides a relatively unified framework of several recent offline or off-policy RL. methods (Haarnoja et al.
2

---

Published as a conference paper at ICLR 2021
2018 Fujimoto et al. 2019 Kumar et al. 2019). It incorporates divergence function between distributions over state- actions in the actor-critic objectives. As with SAC (Haarnoja et al. 2018). one limitation of the algorithm is its sensitivity to reward scale and regularization strength. In our experiments, we indeed observed wide spread of optimal hyper-parameters across different meta-RI. environments, shown in T
To the best of our knowledge, despite attracting more and more attention, the offline meta-RL problem is still understudied. We are aware of few papers that tackle the same problem from different angles (Li et al. 2019b; Dorfman & Tamar 2020) Li et al. (2019b) focuses on specific scenario where biased datasets make the task inference module prone overfit the state-action distributions. ignoring the reward/dynamics information. This so-called MDP ambigu- ity problem occurs when datasets of different tasks do not have significant overlap in their state- action visitation frequencies, and is exacerbated by sparse rewards. Their method MBML requires training of offline BCQ (Fujimoto et al., 2019) and reward/dynamics models for each task, which computationally demanding, whereas method is end-to-end and model-free.
Offline Meta-RL
Dorfman & Tamar 2020) on the other hand, formulate the OMRL as a Bayesian RL (Ghavamzadeh) et al.] 20066) problem and employs a probabilistic approach for Bayes-optimal exploration. Therefore we consider their methodology tangential to ours.
PRELIMINARIES
3.1 NOTATIONS AND PROBLEM STATEMENT
We consider fully-observed Markov Decision Process (MDP) (Puterman) 2014) in determinis- tic environments such as MuJoCo (Todorov al. 2012). An MDP can be modeled M (S,A, P. R. po. 7) with state space S, action space A. transition function P(s'|s, a), bounded re- ward function R(s, a), initial state distribution (s(s) and discount factor 7 (0, 1). The goal is to find policy л (a]ss) to maximize the cumulative discounted reward starting from any state. We introduce the notion of multi-step state marginal of policy æ as 2 (s), which denotes the distribu-
tion over state space after rolling out for č steps starting from state The notation (x(s) denotes the expected reward at state s when following policy ा: Rg (s) = Eass [R(s, a)]. The state-value function ka.k.a. value function) and action-valus function (a.k.a Q-function) are therefore
Vs( = Σ " Exewy (s) [[(se)]
(s, = R(s,a) PBy- (V
(2)
Q-learning algorithms are implemented by iterating the Bellman optimality operator B, defined as: (BQ) a) = R[s, a) + 7 [maxx (s',()
When the state space is large/continuous, Q is used as a hypothesis from the set of function approx- imators (e.g. neural networks).
In the offline context of this work, given a distribution of tasks p(T) where every task is an MDP, we study off- off-policy meta-learning from collections of static datasets of transitions D; {(Si,t; di,t, L Pt.a)lt 1,..,, N) generated by a set of behavior policies (3, (als)) associated with each task index i. key underlying assumption of meta-learning is that the tasks share some com- mon structures. By definition of MDP, in this paper we restrict our attention to tasks with shared state and action space, but differ in transition and reward functions.
We define the meta-optimization objective as ( ET-((T) (e)]
(()) =
(4)
where LT, (0) is the objective evaluated on transition samples drawn from task T, A common choice of ((T) is the uniform distribution on the set of given tasks [Tali = 1,..., n). In this case, the meta-training procedure turns into minimizing the average losses across all training tasks
mmeta = arg min
E (()
(5)
3

---

Published as a conference paper at ICLR 2021
3.2 BEHAVIOR REGULARIZED ACTOR CRITIC (BRAC)
Similar to SAC, to constrain the bootstrapping error in offline RL, for each individual task Ti be- havior regularization (Wu al. 2019) introduces divergence measure between the learner me and the behavior policy T6 in value and target Q-functions. For simplicity, we ignore task index this section:
(s)
Σ E u (6) [Rr - (TToT((-sss)) mo(+ss))]
(6)
1=
= (uu((t))) (-(1)))
where Q denotes target Q-fuctioon without gradients and ò denotes sample-based estimate of the divergence function D. In actor-critic framework, the loss functions of Q-value and policy learning given by, respectively,
Qu Cerstic = E (s.ar,ss)) JwD - a a'mat)l)l'))
(8)
Lactor = B(sans)) (
(9)
3.3 CONTEXT-BASED META-RI.
Context-bait based meta- RL algorithms aggregate context information, typically in form of task specifci transitions, into latent space 2. It can be viewed a special form of RL partially-ohserved MDP (Kaelbling et al. 1998) in which latent representation z as the unobserved part of the state needs to be inferred. Once given complete information of and s combined as the full state, the learning of the universal policy (s(s, z) and value function Vn (s, 2) (Schaul et al. (2015) becomes RL on regular MDP, and properties of regular RL such as the existence of optimal policy and value functions hold naturally. We therefore formulate the context- -baedd meta-RL. problem as solving auggene MDP (TA-MDP). The formal definitions are provided in Appendix
METHOD
Based on our formulation of context-based meta-RL problem, FOCAL first learns an effective rep- resentation of meta-training tasks on latent space 2, then solves the offline RL problem on TA-MDP with behavior regularized actor critic method. We illustrate our training procedure in Figure [ and describe the detailed algorithm in Appendix [A] We assume that pre-collected datasets are available for both training and testing phases, making our algorithm fully offline. Our method consists of three key design choices: deterministic context encoder, distance metric learning on latent space as well as decoupled training task inference and control.
4.1 DETERMINISTIC CONTEXT ENCODER
Similar to Rakelly al. 2019)) we introduce an inference network (o(2(c), parameterized by ø. to infer task identity from context ~ C. In terms of the context encoder design, recent meta-RL methods either employ recurrent neural networks (Duan al. 2016 Wang et al. 2016) capture the temporal correlation, or use probabilistic models (Rakelly et al. 2019) for uncertainties estimation. These design choices are proven effective on-policy and partally-offlin off off-policy algorithms. However, since our approach aims to address the fully-offline meta-RL. problem, we argue that deterministic context encoder works better in this scenario, given a few assumptions:
First, we consider only deterministic MDP in this paper, where the transition function is Dirac delta distribution. We assume that all meta-learning tasks in this paper are deterministic MDPs, which is satisfied by common RL benchmarks such as MuJoCo. The formal definitions are detailed in Appendix B Second, we assume all tasks share the same state and action space, while each characterized by unique combination of transition and reward functions. Mathematically. this means there exists an injective function f T + P x R, where P and R are functional spaces of transition probability P: A {0, 1) and bounded reward R S A IR respectively. A
4

---

Published as a conference paper at ICLR 2021
Context
L (2|c)
gradient
Φ
replay buffer
2
Qw
Coiitic
To (s, z)
Lactor
train tasks
Figure 1: Meta-training procedure. The inference network Is uses context data to compute the latent context variable 2, which conditions the actor and critic, and is optimized by the distance metric learning (DML) objective. The learning of context encoder (Ldml) and control policy (Lactor, feritte) are decoupled in terms of gradients.
stronger condition of this injective property is that for any state-action pair (s, a), the corresponding transition and reward are point-wise unique across all tasks, which brings the following assumption: Assumption (Task- Transition Correspondence). We consider meta-RL with a task distribution p(T) to satisfy task-transitie correspondence if and only if VT1, To p(T), (s, E A:
P1 a) =
= R2(s, a) = T = To
(10)
Under the deterministic MDP assumption, the transition probability function P(-|s, a) is associated with the transition map t x + S (Definition B.3). The task-transition correspondence suggests that, given the action-state pair (s, a) and task T. there exists a unique transition-reward pair (s',r) r). Based on these assumptions, one can define a task-specific map fr A + on the set of transitions Dy:
fT = r.). VT ~ p()) r,) E Df
(11)
Recall that all tasks defined in this paper share the same state-action space, hence ( p(T)) forms function family defined on the transition space S x A x x R, which is also by definition the context space This lends new interpretation that as task inference module, the context encoder (2(c) enforces an embedding of the task-specific map fT on the latent space Z, i.c. As : 8 A 8 x I Z. Following Assumption I every transition {s,, ag sy,r,) corresponds to unique task T. which means in principle, task identity can be inferred from any single transition tuple. This implies the context encoder should be cermutation-invarian embedding of context does not depend on the order of the transitions nor involve any uncertainty. This observation crucial since it provides theoretical basis for few-shot learning (Snell et al.) 2017] Sung et al. 2018) in settings. In particular, when learning in fully-offling fashion, any meta- RL algorithm at test-time cannot perform adaptation by exploration. The thcoretical guarantee that a few randomly-chosen transitions enable effective task inference ensures that FOCAL is feasible and efficient.
and deterministic, since the
4.2 DISTANCE METRIC LEARNING (DML) OF LATENT VARIABLES
In light of our analysis on the context encoder design, the goal of task inference learn a robust and effective representation of context for better discrimination of task identities. Unlike PEARL, which requires Bellman gradients to train the inference network, our insight is disentangle the learning of context encoder from the learning of control policy. As explained in previous reasoning about the deterministic encoder, the latent variable is representation the task properties involv- ing only dynamics and reward, which in principle should be completely captured by the transition datasets. Given continuous neural networks as function approximators, the learned value functions conditioned on latent variable cannot distinguish between tasks if the corresponding embedding vectors are too close (Appendix [ Therefore for implementation, we formulate the latent vari- able learning problem obtaining the embedding do 8 A x R Z of transition data
5

---

Published as a conference paper at ICLR 2021
Inverse squbre
Inverse
Half-Cheetah-Vel
t-NE dimero on
Inverse-square
Inverse
-200
Linear
Average Return
Squarie
t-SNE dimension
SNE dimension 1 Square
-30
Linear
C.NE dimendiion
dimenssion
-00
350
0.5
1-SNE dimension
1-5NE dimension 1
Sample Steps
(a) context embedding
(b) test-task performance
Figure 2: (a) t-SNE visualization of embedding vectors drawn from 20 randomized tasks on Half- Cheetah- Vel. Inverse-power distance metric losses (DML) achieve better clustering. Data points are color-coded according to task identity. (b) FOCAL trained with inverse-power DML losses outperform the linear and square distance losses.
D. = {( as,t, sy rt,)] 1,....NN that clusters similar data (same task) while pushes away dissimilar samples (different tasks) on the embedding space 2, which is essentially distance metric learning (DML. (Sohn) 2016). A common loss function in DML is contrasitive loss (Chopra et al.) 2005 Hadsell al. 2006). Given input data æ4 æj E X' and label y {I, ... L}, it is written
Coont (2g, Zy:9) 1 = y blqs gg1l2 + 1 (3/s * yo hiy.mmax.... m l|q: - 9gll2)²
(12)
where m constant parameter, qi (xz) is the embedding vector 2t. For data point of different tasks/labels, contrastive loss rewards the distance between their embedding vectors by 7,2 norm, which weak when the distance small, in the case when is normalized and qa randomly initialized. Empirically, we observe that objectives with positive powers of distance lead to degenerate representation of tasks, forming clusters that contain embedding vectors multiple tasks (Figure 2ap. Theoretically, this is due the fact that accumulative L loss of distance between data points is proportional to the dataset variance, which may lead to degenerate distribution such Bernoulli distribution. This is proven in Appendix B. To build robust and efficient task inference module, we conjecture that it's crucial to ensure every task embedding cluster to be separated from each other. We therefore introduce negative-power variant of contrastive loss follows:
Lami Cami( Zz:4) q) = 1 = yy 4illā + 1 + y3}3
(13)
q: L
where e >> small hyperparameter added to avoid division by zero, the power n can be any non-negative number. Note that when embedding introduced by Luo et al. (201T). which was proven to better preserve local topology and similarity relationships compared to Laplacian embeddings. We experimented with I (inverse) and 2 (inverse-square) square) this paper and compare with the classical L I metrics in Figure and 5...
2. Equ 13 takes form analogous to the Cauchy graph
5 EXPERIMENTS
In our experiments, we assess the performance of FOCAL by comparing it with several baseline algorithms on meta-RL benchmarks, for which return curves averaged over random seeds. Specific design choices are examined through 3 ablations and supplementary experiments are pro- vided in Appendix
5.1 SAMPLE EFFICIENCY AND ASYMPTOTIC PERFORMANCE
We evaluate FOCAL 6 continuous control meta-environmente of robotic locomotion, 4 of which are simulated via the MuJoCo simulator (Todorov et al., 2012). plus variants of a 2D navigation problem called Robot. Ssparse-Point-Roboe Half-Cheetah- Vel, Salf-Chhetta--wwdaaae

---

Published as a conference paper at ICLR 2021
Ant-Fwd-Bead BBckk and 2 (Point- Point-RRobot-o Wind, Walker-2D-PPe P--ammss)) environments require adaptation by reward and transition functions respectively.
Sparse-
Hull-Chwelah-Vell
Hall-Chaet
Art-Fwwd-Back
1500
-100
1250
V
-200
420
200
3300
502
-400
250
2
-500
-050
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
0.0
0.4
0.5
0.5
Samppe
Sample
Sample
Sample Steps
FOCAL
Batch PEARL
Contestuual BCQ
MENL
Figure 3: Performance vs. number of transition steps sampled for train- ing. Top: Average episodic testing re- turn FOCAL vs. other baselines on 4 meta-environments with different re- ward functions across tasks. Bottom: Average episodic testing return of FO- CAL vs. other baselines on 2 meta- environments with different transition dynamics across tasks.
Point
-3
0.4
0.5
2.5
1.0
0.2
0.4
0/5
Zagss
Sampile Stepn
POCAL
Datch FEATL
Contentoal DC3
MENLL
These meta-RL benchmarks were previously introduced by Finn et al. (2117) and Rakelly et al. (2019), with detailed description in Appendix For data generation, we train stochastic SAC mod- els for every single task and roll out policies saved at each checkpoint to collect trajectories. The offline training datasets are selections of the saved trajectories, which facilitates tuning of the per- formance level and state-action distributions of the datasets for each task.
For OMRI, there are two natural baselines. The first by naively modifying PEARL to train and test from logged data without exploration, which we term Batch PEARI.. The second is Contextual BCQ. It incorporates latent variable z in the state and perform cask-augemented variant of offline BCQ algorithm (Fujimoto et al. 2019). Like PEARL, the task inference module is trained using Bellman gradients. Lastly, we include comparison with the MBML algorithm proposed by Li et al. (019a). Although as discussed earlier, MBML is modell-ased,, two-stage method opposed to our model-free and end-to-end approach, we consider by far the most competitive and related OMRL algorithm to FOCAL, due to the lack of other OMRL methods.
As shown in Figure we observe that FOCAL outperforms other offline meta-RI. methods across almost all domains. In Figure 4b we also compared FOCAL to other algorithm variants including a more competitive variant of Batch PEARL by applying the same behavior regularization. In both trials, FOCAL with our proposed design achieves the best overall sample efficiency and asymptotic performance.
We started experiments with expert-level datasets. However, for some tasks such Ant and Walker, we observed that a diverse training sets result in better meta policy (Table 2 We conjecture that mixed datasets, despite sub-optimal actions, provides broader support for state-action distributions, making it easier for the context encoder to learn the correct correlation between task identity and transition tuples (i.e., transition/reward functions). While using expert trajectories, there might be little overlap between state-action distributions across tasks (Figure 8) which may cause the agent to overfit to spurious correlation. This is the exact problem Li et al. (2019h) aims to address, termed MDP ambiguity. Such overfitting state-action distributions leads to suboptimal latent representa- tions and poor robustness to distribution shift (Table which can be interpreted as a special form of memorization problem in classical meta-learning (Yin et al. 2019). MDP ambiguity problem is addressed in an extension of FOCAL (Li al. 2021)
5.2 ABLATIONS
Based on our previous analysis, we examine and validate three key design choices of FOCAL by the following ablations. The main results are illustrated in Figure and 5
7

---

Published as a conference paper at ICLR 2021
Hal-Cherahhi
E-SHE de merestnn
1-SNE diierrriionn
-20
t-SNE dimension
t-SNE dimension
200
PEARL,
1.SME cimeesion
SINE climenoons
-50
Sampll $0..5
-382 -90 dimension
dimonsion
FOCX (aurt)
Rolck
context embedding
(b) test-task performance
Figure 4: Comparative study of 4 algorithm variants: FOCAL with deterministie/rrobbbbiitiii context encoder, Batch PEARL with/without behavior regularization. (a) t-SNE visualization of the embedding vectors drawn from 20 randomized tasks on Walker-2D-Paramg Data points are color-coded according to task identity. (b) Return curves on tasks with different reward functions (Half-C Maalf-Cheetah-Veie and transition dynamics (Walker-2D-PPrams)
5.2. POWER LAW OF DISTANCE METRIC LOSS
To show the effectiveness of our proposed negative-powe distance metrics for OMRL problem, tested context embedding loss with different powers of distance, from L-2 to L A t-SNE Van der Maaten & Hinton 2008) visualization of the high-dimensional embedding space in Fig- 2a demonstrates that, distance metric loss with negative powers are more effective in separating embedding vectors of different tasks, whereas positive powers exhibit degenerate behaviors, leading less robust and effective conditioned policies. By a physical analogy, the -verre-powwer losses provide "repulsive forces" that drive apart all data points, regardless of the initial distribution. In electromagnetism, consider the latent space as 3D metal cube and the embedding vectors as po- sitions of "charges" of the same polarity. By Gauss's 's law, at equilibrium state, all charges distributed on the surface of the cube with densities positively related to the local curvature of the surface. Indeed, we observe from the "Inverse-square" and "Inverse" trials that almost all vectors are located near the edges of the latent space, with higher concentration around the vertices, which have the highest local curvatures (Figure 7)
To evaluate the effectiveness of different powers DML loss, define metric called effective separation rate (ESR) which computes the percentage of embedding vector pairs of differ- ent tasks whose distance latent space Z is larger than the expectation of randomly distributed vector pairs, i.e., 221/3 on (-,, Table ! demonstrates that DML losses of nega- tive power are more effective in maintaining distance between embeddings of different tasks, while no significant distinction is shown in terms of RMS distance, which is aligned with our insight that RMS or effectively classical 1.2 objective, can be optimized by degenerate distributions (Lemma B.T. This is the core challenge addressed by our proposed inverse-powee pporee loss.
Table 1: Embedding Statistics on Vel (latent space di- mension 5).
Loss Inverse-square
RMS
ESR
1.282
0,861
Inverse
1.217
0.840
Linear
1.385
0.819
Square
1.415
0.506
5.2.2 DETERMINISTIC VS. PROBABILISTIC CONTEXT ENCODER
Despite abundance successes of probabilistie/varitioona inference models in previous work (Kingma & Welling 2013 Alemi et al. 2016 Rakelly et al.) 2019), by comparing FOCAL with deterministic and probabilistic context encoder Figure/4) we observe experimentally that the for- mer performs significantly better on tasks differ in either reward or transition dynamics in the fully offline setting. Intuitively, by our design principles, this is due to
Offline meta-RL. RL does not require exploration. Also when Assumption | is satisfied, there is not need for reasoning about uncertainty during adaption.
8

---

Published as a conference paper at ICLR 2021
FOCAL (ound)
FOCAL coupled
DONE dimension
N-NNC cimension
]
(a) context embedding
(b) test-task performance
Figure 5: FOCAL vs. FOCAL with coupled gradients. (a) t-SNE visualization of the embed- ding vectors drawn from 20 randomized tasks on Walker-2D-Params. Data points are color-coded according to task identity. (b) Return curves Walker-2D-Params.
2. The deterministic context encoder in FOCAL is trained with carefully designed metric- based learning objective, detached from the Bellman update, which provides better effi- ciency and stability for meta-lcarning.
Moreover, the advantage of our encoder design motivated by Assumption I is also reflected in Figure 4a our proposed method the only variant that achieves effective clustering of task embeddings. The connection between context embeddings and RL performance is elaborated in Appendix C
5.2.3 CONTEXT ENCODER TRAINING STRATEGIES
The last design choice of FOCAL is the decoupled training of context encoder and control pol- icy illustrated in Figure [ To show the necessity of such design, in Figure 4 we compare our proposed FOCAL with a variant by allowing backpropagation of the Bellman gradients encoder. Figure 5a that our proposed strategy achieves effective clustering of task context and therefore better control policy, whereas training with Bellman gradients cannot. As consequence, the corresponding performance gap is evident in Figure 5b We conjecture that on complex tasks where behavior regularization is necessary to ensure convergence, without careful tuning of hyper- parameters, the Bellman gradients often dominate over the contribution of the distance metric loss. Eventually, context embedding collapses and fails to learn effective representations.
context
Additionally however, we observed that some design choices of the behavior regularization, partic- ularly the value penalty and policy regularization in BRAC (Wu et al., (2019) can substantially affect the optimal training strategy. We provide more detailed discussion in Appendix E.2
CONCLUSION
In this paper, we propose novel fully-offline meta-RL. algorithm, FOCAL, in pursuit of more practi- cal RL. Our method involves distance metrie learning of deterministic context encoder for efficient task inference, combined with an actor-critic apparatus with behavior regularization to effectively learn from static data. By re-formulating the meta-RL tasks task-augmented MDPs under the lask-transition correspondence assumption, we shed light on the effectiveness of our design choices in both theory and experiments. Our approach achieves superior performance compared to existing diverse set of continuous control meta-RI. domains. Despite the success, the
OMRL algorithms strong assumption we made regarding task inference from transitions can potentially limit FOCAL's robustness to common challenges in meta-RI. such as distribution shift, sparse reward and stochastic environments, which opens up avenues for future work of more advanced OMRL algorithms.
7 ACKNOWLEDGEMENTS
The authors grateful to Yao Yao, Zhicheng An and Yuanhao Huang for rumning part of the base- line experiments. A special thank to Yu Rong and Peilin Zhao for providing insightful comments and being helpful during the working process.

---

Published as a conference paper at ICLR 2021
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. International Conference on Machine Learning (ICML), 2020.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy, Deep variational information bottleneck. arXiv preprint arXiv:1612.004100 2016.
Jonathan Binas, Leonie Luginbuehl, and Yoshua Bengio. Reinforcement learning for sustainable agriculture. In JCML 2019 Workshop Climate Change: How Can Al Help, 2019.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'0S) 05), volume
pp. -46..56..66 IEEE, 2005.
Ron Dorfman and Aviv Tamar. arXiv:2008. 02598, 2020.
Offline meta reinforcement
learning.
arXiv preprint
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI:: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:6611.0277, 2016.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J Smola. Meta-q-learing. arXiv preprint arXiv:1910.00125: 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400 2017.
Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 0052-2062, 2019.
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning: A survey. arXiv preprint arXiv:1609.044466 006600900446,,... 2016.
Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Doshi-Velee and Leo Anthony Celi. Guidelines for reinforcement learning in healthcare. Nat Med, 25(1):16-18, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- tion (CVPR 06), volume 2, pp. 1735-1742. IEEE, 2006.
Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Residual reinforcement learning for robot control. In 2019 International Conference on Robotics and Automation (ICRA), pp. 6023- 6029. IEEE, 2019.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-13, 1998.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes 1rXiv: 14, 2013.
arXiv preprint
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. /CML deep learning workshop, volume 2. Lille, 2015.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Sys- tems, pp. 1 11784-1 11794, 2019.
10

---

Published as a conference paper at ICLR 2021
Hongyang Li, David Eigen, Samuel Dodge. Matthew Zeiler, and Xiaogang Wang. Finding task- relevant features for few-shot learning by category traversal. In Proceedings of the IEEE Confer- on Computer Vision and Pattern Recognition, pp. 1-10, 2019a.
Jiachen Li, Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Henrik Iskov Christensen, and Hao Su. Multi-task batch reinforcement learning with metric learning. arXiv, pp. arXiv-1909, 2019b.
Lanqing Li, Yuanhao Huang, and Dijun Luo. Improved context-based offline meta-rl with attention and contrastive learning. arXiv preprint (rXiv:2102.10774, 2021.
Dijun Luo, Chris HQ Ding, Feiping Nie, and Heng Huang. Cauchy graph embedding. In ICML,, 2011.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprint 1707.00707.033 2017.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcemennt learning via probabilistic context variables. In International conference on machine learning. pp. 5331-5340, 2019.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine leurning, pp. 1312-1320. PMLR, 2015.
Shai Shalev-Shwartz Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint 1xxiv:1610.03325,,0000 2016.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked: Be- havioral modelling priors for offline reinforcement learning. arXiv preprint NXix:2000.088900 2020.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lueas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 355--359,, 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp- 4077-4087, 2017.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, pp. 1857-1865, 2016.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference Computer Vision and Pallern Recognition, pp. 1199-1208 208, 2018
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning Artificial intelligence review, 18(2):77-95, 2002.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350-3344 2019.

---

Published as a conference paper at ICLR 2021
Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn arXiv preprint arXiv:1611.05763, 2016.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911:1911,,1311,, 2019.
Liu Yang and Rong Jin. Distance metric learning: A comprehensive survey. Michigan State Uni- versiy, 2(2):4, 2006.
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforce- ment learning. In AAAI, pp. 6672-6679, 2020.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine. and Chelsea Finn. Meta-learning without memorization, arXiv preprint arXhv:1912.03820, 2019.
12

---

Published as a conference paper at ICLR 2021
Appendices
A
PSEUDO-CODE
Algorithm 1: FOCAL Meta-training
Given:
Pre-collected batch D; {(vi.) on o rislba... of set of training tasks (Te) 1.....
drawn from p(7)
Learning rates Cr1, 022, 03
Initialize context replay buffer C; for each task Ti
Initialize inference network go(2(c), learning policy my (als, 2) and Q-network Qu 2, a) with
parameters 6, ! and vb
while not done do
for each T, do
for t= 0, T do Sample miti-batches of B transitions {(S ai,, & Ti,)aa. Dr and update C₁
end end
Sample mini-batches of M tasks p(T) for step in training steps do for each T, do
10
11
Sample mini-batches and b; C₁ for context encoder and policy training for each T, do
13
Sample
ej from C2
Cij = (Cm Cji q)
15
16
end
L Dactor Lactor (bs, ((;)) Li Coitii Loritielle, ((ei))
17
18
19
end
② - Φ a1 V Eij Lim
20
0 0 0: Vo factor
+ a3 V 4 C
end
24 end
Algorithm 2: FOCAL Meta-testing
Given:
Dasa... of a set of testing tasks Pre-collected batch D. = (SV ,, a , (e g
[Tr drawn from p(T)
Initialize context replay buffer Cv for each task Te
for each Ty do
for 0, T - do
(e D. and Sample mini-batches of B transitions ey update C,+
Compute 2µ (c(c))
Roll out policy To(a|s, zx) for evaluation
end
end
13

---

Published as a conference paper at ICLR 2021
B DEFINITIONS AND PROOFS
Lemma B.1. The contrastive loss of a given dataset X' = { = 1,...,N, NN is proportional to the variance of the random variable X X
Proof. Consider the contrastive loss Eit -xy)² which consists of N(N -1) pairs of different samples (ry) drawn from X. It can be written as
Σ (F)
(a. x,)² =2 (N (N-1)
Σ
Σ
(14)
The variance of X ~ X' is expressed as
Var(X) = (X - X)2
(15)
= X (x))
(16)
1 Σ x
2
(17)
N
N
(N - 1)
Σ Σ
(18)
N2
(
where X denotes the expectation of X. By substituting Equ 18 into
14
we have
Σ ドイj
[x -,,,² 2N2 (Var(X)
(19)
Definition B.1 (Task-Augmented MDP). A task-augmented Markov Decision Process (TA-MDP) can be modeled as M (S, 2, R, Po, 7) where
S: state space
Z: contextual latent space
A: action space
P: transition function P(s/,z//e z, a) (s'(s,,, a.) if there is no intra-task transition
R: reward function R(s, z, a) = R (s(s.a) a)
(s z): joint initial state and task distribution
Y E (0, 1): discount factor
Definition B.2. The Bellman optimality operator B- on TA-MDP is defined as
(B.Q)(s, R(s, a) + (st, (m
(20)
Definition B.3 (Deterministic MDP). For deterministic MDP, transition map t x S exists such that:
P(s'ls,a) 8(5' - ((s,a))
(21)
where 5(x= - y) is the Dirac delta function that is zero almost everywhere except = y.
14

---

Published as a conference paper at ICLR 2021
C IMPORTANCE OF DISTANCE METRIC LEARNING FOR META-RL ON TASK-AUGMENT MDPs
We provide informal argument that enforcing distance metric learning (DML.) is crucial for meta- RL on task-augmented MDPs (TA-MDPs). Consider a classical continuous neural network Ng parametrized by 9 with N layers, ny IN many nodes the (-th hidden layer for ( = .... L, input dimension Yoo, output dimension R4+1 and nonlinear continuous activation function o R +
R. can be expressed as
No(w) 0 Ar.
(1)))
(22)
where A 1 + R is an affine linear map defined by A,(x) = W2 + by for x f4-1 dimensional weight matrix Wi and ny dimensional bias vector b, and 01 _ + R an element- wise nonlinear continuous activation map defined by (1(2) := affine and activation map is continuous, their composition No is also continuous, which means by definition of continuity:
(T,)))) Since every
Vc > 0, Bn > 0 s.t.
(23)
x2| ア (N((1)) No(x2) <
(24)
where |-| in principle denotes any valid metric defined on Euclidean space Ro A classical example is the Euclidean distance.
Now consider No the value function TA-MDP with deterministic embedding, approximated by a neural network parameterized by 0:
Qs(s,a,2) = Qo(s, a, (s, + ( [Vo
(25)
The continuity of neural network implies that for pair of sufficiently close embedding vectors (2t, 2j), there exists sufficiently small and > 0 that
21, 22 3, |21 221
(Qo(s,a,21) 20()) <E
(26)
Eqn 26 implies that for a pair of different tasks (Ts, T,) ~ p(7), if their embedding vectors are sufficiently close in the latent space Z. the mapped values of meta-learned functions approximated by continuous neural networks are suffciently close too. Since by Equ 25 due to different transi- tion functions Ps. (s'l ss a). Pz, (s'ls, a) and reward functions R2 (s, a). R. (s, a) of (T,, T,). the distance between the true values of two Q-functions (Qo(s, =;) Qo 2,)1 is not guaranteed be small. This suggests that meta-RI. algorithm with suboptimal representation of context em- bedding (c), which fails in maintaining effective distance between two distinct tasks T. Tj. unlikely to accurately learn the value functions (or any policy-related functions) for both tasks simultaneously. The conclusion can be naturally generalized to the multi-tast meta neta-RL setting.
15

---

Published as a conference paper at ICLR 2021
D EXPERIMENTAL DETAILS
D.1 DETAILS OF THE MAIN EXPERIMENTAL RESULT (FIGURE AND
The main experimental result in the paper is the comparative study of performance of FOCAL and three baseline OMRL algorithms: Batch PEARL, Contextual BCQ and MBML, shown in Figure 3 Here in Figure 6 we plot the same data for the full number of steps sampled in our experiments. Some of the baseline experiments only lasted for 10 steps due to limited computational budget, but are sufficient to support the claims made in the main text. We directly adopted the Contextual BCQ and MBML implementation from MBML's official source code perform the experiments on our own dataset generated by SAC algorithm The DML loss used in experiments in Figure inverse-squared, which gives the best performance among the four power laws we experimented with in Figure 2
Sparse-Point-Rooat
Cheetah Vel
Cheetah
1500
12
-100
1250
-200
1000
750
Average
6-
-300
-400
250
2
-500
0-
-250
10a
10
10a
10
10a
10
Sample Steps
Sample Steps
Sample Steps
Ant-Fwd-Back
Walker-2B Salker-2D-Params
Point-Robot-Wing
-4
600
-5
Wwwwyy
400
300
-6
200
200
-7
0
-200
100
-8-
-400
0
-600
10*
10a
10a
10a
10a
10a
10a
Sample Steps
Sample Steps
Sample Steps
FOCAL
Batch PEARL
Contextual BCQ
MBML
Figure 6: Average episodic testing return of FOCAL other baselines on five meta-environments
In addition, provide details the offline datasets used produce the result. The performance levels of the training/testing data for the experiments are given in Table which are selected for the best test-time performance over four levels: expert, medium, random, mixed (consist of all logged trajectories of trained SAC models from beginning (random quality) to end (expert quality)). For mixed data, the diversity of samples is optimal but the average performance level is lower than expert. A summary of the fixed datasets used for producing 3
given in Table
Lastly, shown in in Figure 7 we also present faithful 3D projection (not processed by t-SNE) of la- tent embeddings in Figure 4a| Evidently, our proposed method is the only algorithm which achieves effective clustering of different task embeddings. As validation of our intuition about the analogy between the DML loss and electromagnetism discussed in 5.2. the learned embeddings do clus-
hhttp:://ithub. huttsss//dLihua..cmm//eeeeeeen /ttp///ttaaaaaia
Forr sparse reward environments like Sparse-Poin Painse-Poiit---ee Robot, we observed no adapation at test time (retum stays zero) for both Contextual BCQ and MBML, which might be due to incorrect implementation or just that both algorithms fail to adapt in sparse reward scenarios. To avoid drawing conlusion too hastily, we chose to present Contextual BCQ and MBML result for Sparse-Point-Roooe
the moment.
16

---

Published as a conference paper at ICLR 2021
Table Quality of data used for best test-time performance. We maintain the same quality of data for training and testing due to algorithm's 's sensitivity to distribution shift. From our experiments, we observe that for some envs/tasks, datasets with the best performance generate the best testing result, whereas for some envs/tasks, the diversity of data matters the most.
Meta Env
Training Data
Testing Data
Sparse-Point--E Half- Selff-Cheetah-Ae Vel
expert
expert
expert mixed
expert mixed mixed mixed expert
Ant-Fwd-Baad wwdd-Back
Walf-Cheetah-Fwee Fanthh--wd--kkkk
mixed
Walker-2DD-Param
mixed
expert
Table 3: Details the fixed datasets used for producing Figure and 6 "Checkpoints" column stand for starting epoch: ending epoch: checkpoint spacing.
The three numbers in the
tasks
Checkpoints
trajectories
Trjectory Steps
Sparsc-Point-Robee
100
2300:4800:200 200
51
200
2
3IG
Half-Cheetah-
50000:950000:
Ant Fowd. Back
2
10000:790000: 10300)
55
200
27
1.IG
Chll-Cheetah Pwd-Back
20
Walker 2D.Param Params
50
500000950000:550000
53
200
17
4,60
ter around the corners and edges of the bounded 3D>-pojeected latent space, which are locations of highest local curvatures.
FOCAL (ours)
FOCAL, probabilistic
1.00
1.00
0.75
0.75
0.50
025
0.5
0.00
0.00
-0255
-0.00
-0.00
0075
1.00
14-4
00 75 25.0
0,200
dii 2
dim 2
-2 dim
dim
-1.00
2
Batch PEARL
Batch PEARL, BRAC
dim 3
dim 3
te-5
16-5
0
dim 2
0 20
dim 2
Ie-5
-
-2
1e-5
-2 0
dim
dim
Figure 7: 3D projection of the embedding vectors 1- drawn from 20 randomized tasks on Walker-2D-Pardd Params. Data points are color-coded according to task identity.
17

---

Published as a conference paper at ICLR 2021
D.2 DESCRIPTION THE META ENVIRONMENTS
Sparse-Point-Robul A 2D navigation problem introduced in PEARL (Rakelly et al, 2019). Starting from the origin, each task is to guide the agent to specific goal located on the unit circle centered at the origin. Non-sparse reward is defined as the negative distance from the current location to the goal. In sparse-reward scenario, reward is truncated O when the agent outside a neighborhood of the goal controlled by the goal radius. While inside the neighborhood, agent rewarded by distance each step, which is positive value.
Point-Robot- Wind: variant of Sparse- Poerse-Poont--eee Roint-Robot. Task differ only in transition func- tion. Each task is associated with the same reward but distinct "wind" sampled uniformly from [-1,1]². Every time the agent takes step, drifts by the wind vector. We use / 0.05 in this paper.
Buals-hheetab-F---BBeee
Control Cheetah robot to move forward or backward. Re-
ward function is dependent on the walking direction.
Half-Cheetah- Vel: Control Cheetah robot to achieve target velocity running forward.
Reward function is dependent on the target velocity.
aat-Fwd-Backs Control an Ant robot to move forward or backward. Reward function is dependent on the walking direction.
Balker-2D-Paranee Agent is initialized with some system dynamics parameters random- ized and must move forward, vironments since tasks differ in transition function. Transitions function is dependent randomized task-specific parameters such as mass, inertia and friction coefficients.
unique environment compared to other MuJoCo en-
D.3 HYPERPARAMETER SETTINGS
The details of important hyperparameters used to produce the experimental results the paper are presented in Table and
Table 4: Hyperparameters used to produce Figure 3 Meta batch size refers to the number of tasks
used for computing the DML loss C at time. Larger meta batch size leads to faster convergence but requires greater computing power.
Hy Hyperperameees
Spurap-Font-
Foim-Rohee Wind
Hal--Chetah--e VOT
Amt-Pin-Back
Fall Bhcetall Sac-saa---uil-Baa Ponl-Back
Waller-2--Paaoos
reward scals
100
100
5
DML lass weight!:))
1
10
10
helovioe regulariationn stengther)
0
50
1d5
500
50
value (in BRAC)
NA
N6A
Fube
True
Fulse
butter (per
124
164
164
butch
256
256
256
256
butch
divergence discrimiattor
actor Jnfrrl
Jrien))
163
16-3
discount fastor
0.9
0.99
0.99
0.99
0.99
training tarkss
40
80
20
lesting tasks
10
20
5
pooi ridius
02
N/A
N/A
N/A
N/A
N/A
Intent space dimension
5
20
20
20
network witth coontest mucode)
200
200
200
200
200
200
actwork depth (oomost enender)
3
3
3
network width (otber))
300
300
300
300
300
300
nawerk depth (others)
3
3
maximum qpisode length
20
200
200
200
200
18

---

Published as a conference paper at ICLR 2021
Table 5: Hyperparameters used to produce Figure 2a
(a) Compared to Half-Cheetah-V Vel experiment T latent space dimension were reduced to speed up computation. Also the value penalty
The DML. loss weight 3 and coefficient (defined Eqn 13 used in experiments of Figure 2a to match the scale of objective functions of different power laws. The weights chosen such that all terms are equal when the average distance x; and 23 per dimension is 0.5, reasonable value given
(b)
used in behavior regularization.
Hyperparaneters
Hall Cheetah-
reward scale
behavior regularization strength(n)
500
value penalty (in BRAC)
True
buller size (per task)
1c4
x (-1,1))
batch size
256
meta batch size
Trials Inverse-Square
B
Ir[[ divergence discrimmintoor dml Ir(aa)
le.4
le-3
0.1
actor_lrfag)
le.3
Inverse
2
0.1
critic In(og)
le-3
discount factor
0.99
Linear
8
0.1
# training tasks
Square
16
0.1
testing tasks latent space dimension
20
network width (context encoder)
200
network depth (context encoder)
netwark width (others)
300
network depth (others)
maximum episode length
200
19

---

Published as a conference paper at ICLR 2021
E ADDITIONAL EXPERIMENTS
E. SENSITIVITY TO DISTRIBUTION SHIFT
Since in OMRL, all datasets are static and fixed, many challenges from classical supervised learning such as over-fitting fitting exist. By developing FOCAL, we are also interested its sensitivity distribu- tion shift for better understanding of OMRL algorithms. Since for each task T,, our data-generating behavior policies (a(s) are trained from random expert level, we select three performance lev- els (expert, medium, random) of datasets to study how combinations of training/testing sets with different qualitiesdisttibutions affect performance. An illustration of the three quality levels on Prisse-Point-Reee is shown in Fig
random
medium
expert
C6
62-
22-
37
07-
34
00
28
04 32
Figure 8: Distribution of rollout trajectories of trained SAC policies of three performance levels: random, medium and expert. Since reward is sparse, only states that lie in the red circle are given non-zero rewards, making meta-learning more challenging and sensitive to data distributions.
Table 6: Average testing return of FOCAL Sparse- Laiiee-Pontt-Reee tasks with different quali- ties/distributions of training/testing sets. The numbers in parenthesis are the performance drop due distribution shift (compared to the scenario where the testing distribution equals the training dis- tribution).
Training expert medium
Testing
FOCAL
expert
8.16(-)
medium
8.44(-)
random random
2.34(-)
expert
medium
(.12
expert
random
(.43(3.73)
medium expert
8.25(0.19)
medium
random
6.76(1.68)
Table shows the average return test-time for various training and testing distributions. Sensitiv- ity to distribution shift is confirmed since training/testing on the similar distribution of data result in relatively higher performance. In particular, this significant sparse reward scenario since Assumption is no longer satisfied, With severe over-fitting and the MDP ambiguity problem elab- orated in the last paragraph of $5.1 performance of meta-RL policy is inevitably compromised by distribution mismatch between training/testing datasets.
E.2
VALUE PENALTY AND POLICY REGULARIZATION IN BRAC
Discussed in (32 BRAC (Wu et al. 2019 introduces possible regularization in the value/Q-ffunction
(Eqn 6 and therefore the critic loss (Eqn 8), well in the actor loss (Eqn If regularization applied on both or only on the policy, it is referred to as value penalty and policy regularization respectively. In the BRAC paper, Wu al. (2019) performed extensive tests and concluded that the two designs yield similar performance, with value penalty being slightly better overall. Since BRAC OMRL setting. In general,
designed for single-task offline RI., again tested both
20

---

Published as a conference paper at ICLR 2021
POCAL lours)
POCAL, caopled
1-SRE dimersionn
-15 -30
dimeraion
60
ROCAL cospled
context emhedding
(h) test-task performance
Figure 9: FOCAL vs. FOCAL with coupled gradients and policy regularization. The task representation alone of the coupled training scheme might not be superior, but the policy per- formance can be improved due to end-to-end optimization. (a) t-SNE visualization of the embed- ding vectors drawn from 20 randomized tasks on according to task identity. (b) Return curves Walker-2D- P--aaams...
Data points are color-coded
found that on complex tasks such Ant, value penalty usually requires extremely large regulariza- tion strength (Table 4 to converge. Since the regularization is added to the value//-functton, this results in very large nagative Q value (Figure 10) and exploding Bellman gradients. In this scenario, training the context embedding with backpropogated Bellman gradients often yields sub-optimal la- tent representation and policy performance (Fig 5), which leads our design of decoupled training strategy discussed in 5.2.3
For policy regularization however, the learned value/Q-functionn approximates the real value (Figure Iaj, leading to comparable order of magnitude for the three losses Ldmi, Lactor and Ceritie- In this case, decoupled training of context encoder, actor and critic, may give competitive or even better performance due to end-to-end optimization, shown in Figure
E.3 DIVERGENCE OF Q-FUNCTIONS IN OFFLINE SETTING
The necessity of applying behavior regularization on environment like Ant-Fwd-Bace and Walker- 2D>-Params to prevent divergence of value functions is demonstrated in Figure TO and
Ant-Twd--eed WedBk
2e11
Antt wwwd--iikkk
-04
cro Seen
-0.6
-12
-1.4
0.0
05
50
1.0
1.5
2.0
2.5
Sample
(a) FOCAL, value penalty
(b) Batch PEART.
Figure 10: FOCAL with value penalty vs. Batch PEARL on Ant-Fwd-Back. The Q-function learned by Batch PEARL diverges (> 10"1) whereas the Q-function of FOCAL, despite its large order of magnitude due to value penalty, converges eventually given proper regularization (a 10%)
21

---

Published as a conference paper at ICLR 2021
Maikker-2-Paaammme FOCAL, policy regularization
160
Walker20-PParams
400
Batch PEARL
302
1.5
220
1.0
0.5
Sample Stezs
165
Sample
185
(a) FOCAL, policy regularization
(b) Batch PEARL
Figure 11: FOCAL with policy regularization vs. Batch PEARL on Walker-22 Miikerr--D-aaee 2D-Params. The Q- function learned by Batch PEARL diverges (>> 10a) whereas the Q-function of FOCAL, converges to the true discounted cumulative return (+s 200 for 7 = 0.99).
22

---

Published as a conference paper at ICLR 2021
F IMPLEMENTATION
We build our algorithm on top of PEARL and BRAC, both are derivatives of the SAC algorithm. SAC an off-policy actor-critic method with a maximum entropy RL objective which encourages exploration and learning stochastic policy. Although exploration is not needed in fully-offline scenarios, we found empirically that maximum entropy augmentation is still beneficial for OMRL, which is likely due to the fact that in environments such as Ant, different actions result in same next state and reward, which encourages stochastic policy.
All function approximators in FOCAL are implemented as neural networks with MLP structures. For normalization, the last activation layer of context encoder and policy networks are invertible squashing operators (tanh), making 2 bounded Euclidean space (-1,1), which is reflected in Figure
As in Figure [ the whole FOCAL pipeline involves three main objectives. The DML loss for training the inference network qa (=(c) is given by Eqn 13 for mini-batches transitions drawn from training datasets: æ; Di, æj Dy. The embedding vector qt. qj computed as the average embedding over Vy and zj. The actor and critic losses are the aagm-nngeented version of Eqn (8 and 9
Leitie B( (rr ,,a. Qv(s, 2))) a'maa(-1a")
(27)
Lactor - B(s,aar (Q a")] aD]
(28)
where Q is target network and E indicates that gradients not being computed through it. As discussed in (Kumar al., 2019; Wu, Tucker, and Nachum, 2019), the divergence function ò can take form of Kernel MMD (Gretton et al., 2012), Wasserstein Divergence (Arjovsky, Chintala, and Bottou, 2017) f-divergences (Nowozin al., 2016) such as KL divergence. In this paper, we use the dual form (Nowozin, Cseke, and Tomioka, 2016) of KL divergence, which learns a discriminator g with minimax optimization to circumvent the need of a cloned policy for density estimation.
In principle, as core design choice of PEARL, the context used to infer ( c) can be sampled with different strategy than the data used compute the actor-critic losses. In OMRL however, we found this treatment unnecessary since there is no exploration. Therefore training of DML and actor- critic objectives are randomly sampled from same dataset, which form an end-to-end algorithm described in Algorithm] I andd]
23


Reviews:
[{"review_id": "Jt42PVBWdKp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of offline meta reinforcement learning, where an agent aims to learn a policy which can adapt to an unseen task (dynamics/reward), but from entirely offline data. As a result of being fully offline, the agent can no longer explore in the new task at test time, but instead receives randomly sampled transitions from the new task, from which it must infer the task. They then propose a method for learning task inference from fully offline data, as well as a policy conditioned on this task encoding from offline data built off behavior regularized actor critic (BRAC). Results indicate that in this outperforms PEARL as well as multi-task offline RL with BCQ.  \n\nPros:\nThe problem this paper tackles is important, and is formulated in a practical way. While in general meta-RL is often closely tied to online exploration to infer what the task is, in practice it seems like having an agent be able to adapt to a new task given just a few transitions from a potentially different policy would be a valuable instantiation of meta-RL. It also integrates nicely with progress in batch RL. \n\nSecond, the idea of decoupling the task inference objective using the metric loss does nicely integrate with the setting of using an offline dataset, and seems to work well as shown by the qualitative examples. While it does depend heavily on having access to the task labels during training, this assumption is used in a lot of meta-RL work. Additionally, the proposed negative power variant of the metric loss does seem to better encourage separation of tasks with different task labels, and seems like a good tool for metric learning in general.\n\nLastly, the experiments do compare against the relevant baselines to the best of my knowledge, and on tasks similar to those used in prior works. The ablations are also interesting, and make a compelling case for why in the offline case deterministic context encoders may be more fitting (when the tasks satisfy Assumption 1).\n\nCons:\nIn general I think the key assumption in this work, that tasks can be inferred from an unordered set of transitions is limiting, especially for tasks with sparse reward, or which involve adapting to changing dynamics. However, this does seem like it is the best that one can do while being totally offline, any more complex tasks would require some form of online exploration.\n\nI'm not convinced of the argument in Appendix C, that the reason why using the control policy to learn the context encoder works poorly is due to neural networks inability to tell apart small differences in the embedding vectors. Rather I think the decoupled training allows for an easier optimization of the task embedding, and then the policy can be learned using good (and stationary) task embeddings throughout all training. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "DaTzPKtkzC", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thanks for the feedback. However, I think most of the performance gain came from behavior regularization, which is an already existing technique, rather than the proposed metric learning method (Figure3 and Figure 5). I’ll keep my ratings unchanged.\n\n============\n\nSummary: \nThe paper presents a mode-free, end-to-end offline meta-reinforcement learning algorithm.\nThe authors proposed the context encoder to encode the task from the history of transitions (called the “context”) using metric-based approaches. Specifically, they embedded the context to a latent vector z by clustering similar data points (context) taken from the same task while pushing away dissimilar data points taken from different tasks. The latent context variable z is used to condition the actor and the critic.\nExperiments are performed on the MuJoCo simulator and the 2D navigation problem with a sparse reward called Sparse-Point-Robot, using different reward functions depending on the task. The method proposed, FOCAL (Fully-Offline Context-based Actor-critic meta-rL algorithm), outperforms other offline meta-RL algorithms.\nThe main contribution of the paper comes from combining the context-based approach with metric learning.\n \nPros: \n- Overall, the paper is well written. The method of using metric learning has been well-motivated by toy experiments on the context embedding. \n- The paper tackles offline reinforcement learning and meta-learning, which are emerging topics in RL/ML.\n \nCons:\n- Format: The paper violated the paper length limit of 8 pages for the main context. \n- Missing experiments: The paper directly followed the algorithm and the experiment setting of “Efficient Off-Policy Meta-RL” (Rakelly et al., 2019) while omitting experiments on the MuJoCo walker and humanoid environments. Also, for the walker-2d experiment, only the toy experiment on context embedding is available with no return curve.\n- Ablations: The authors compare their method (FOCAL) to Batch PEARL while describing the Batch PEARL as “a vanilla version of FOCAL without behavior regularization or distance metric learning.” It would be interesting to compare FOCAL and Batch PEARL both with the behavior regularization to observe the effect of metric learning in isolation. \n- Novelty: The algorithm in Appendix A seems incremental to the algorithm in “Efficient Off-Policy Meta-RL” (Rakelly et al., 2019). The only difference is that the setting is offline and using metric learning to encode the context instead of KL-divergence. Although the combination of context encoding and metric learning is interesting, the proposed is method is still incremental. \n\n\nMinor comments:\n- There was no source code when I followed the Github link provided in the paper.\n- In the return curves on Half-Cheetah-Vel and Ant-Fwd-Back in Figure 3, there is no value near 1e6 sample steps, where the deterioration of performance begins.\n- I understood that all tasks share the same state and action space, with different transition and reward functions. But all the experiments in the paper use the same transition. I wonder if there is an experiment with a different transition depending on the task.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AIwEl3yyx9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary : \nThe paper studies meta-reinforcement learning in the fully offline setting, and proposes a novel algorithm 'FOCAL'. Given offline datasets for tasks sampled from some prior, the algorithm learns a context encoder using distance-based metrics. The encoder is used for inferring the task-latent $z$, which is used to condition the policy rollout $\\pi(a | s, z)$. They demonstrate experiments where FOCAL outperforms baselines like PEARL.\n\nReasons for score :\nWhile I concur with the motivations of the paper, I vote for rejecting the paper (marginally below acceptance threshold). My main concern is the limited novelty of the proposed solution and some missing ablations (c.f weaknesses). I encourage the authors to incorporate feedback for the reviewers and work towards a stronger submission.\n\nStrengths:\n+ The paper is well written and quite easy to follow. The problem is sufficiently well motivated, and the algorithm builds on previous approaches, particularly PEARL/BRAC.\n+ The proposed algorithm seems to outperform reasonable baselines on few navigation/Mujoco tasks in the offline setting.\n\nWeaknesses:\n- As the authors identify, the assumption of task-identifiability from any transition (s, a) seems infeasible in a practical setting. This significantly reduces the scope of the algorithm. The novelty of the proposed algorithm is also quite limited it builds on the framework of PEARL by using (i) distance-based metrics for task inference, which have been studied in the multi-task literature (ii) offline policy learning, where it uses off-the-shelf baselines from the offlineRL literature.\n- The authors include few ablations, I believe adding one/more of the following would improve the quality of discussion:\n       (i) Evaluating how well the context-encoder performs in inferring the meta-test tasks? \n       (ii) Ablating the choice of encoder : Comparing the performance of BatchPEARL, where the encoder is still learned with Bellman gradients, while the policy is behaviorally regularized. I consider this to be a stronger baseline than the existing version of BatchPEARL.\n- Experiments pertaining to 5.2.2 in the ablations are not well-motivated. Given the contrastive loss definition in equation(13), it follows that the “inverse-power” losses learn reasonable task embeddings. In my understanding, n>0 here invalidates the choice of “linear”/”square” objectives. I’d be happy to be corrected if I misunderstood the setup. \n- The authors are encouraged to add details sufficient to reproduce the results. In particular, the setup of the baseline “ContextualBCQ” is scant on details.\n\nMiscellaneous:\n- The paper has several minor grammatical errors which could be corrected in the final draft.\n- I would encourage the authors to add more discussion to the experiments section. The current version reflects the empirical results, but provides little insight into them. Last half of section 5.1 has some mention on the importance of choice of datasets, but doesn’t provide details (until in the Appendix).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Meta-learning for offline RL is an understudied topic with lots of potential impact in the research community. This paper takes the first stab against that challenging problem by proposing a solution similar based on PEARL and distance metric learning. The results look good and it seems like the authors have addressed some of the concerns raised by the reviewers. As a result, I suggest to accept this paper.

However, this paper still has some shortcomings as reviewers suggested more baselines with more experiments on standardized benchmarks such as D4RL or RL Unplugged could make the paper stronger.
Decision: Accept (Poster)