FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization

1 INTRODUCTION . Applications of reinforcement learning ( RL ) in real-world problems have been proven successful in many domains such as games ( Silver et al. , 2017 ; Vinyals et al. , 2019 ; Ye et al. , 2020 ) and robot control ( Johannink et al. , 2019 ) . However , the implementations so far usually rely on interactions with either real or simulated environments . In other areas like healthcare ( Gottesman et al. , 2019 ) , autonomous driving ( Shalev-Shwartz et al. , 2016 ) and controlled-environment agriculture ( Binas et al. , 2019 ) where RL shows promise conceptually or in theory , exploration in real environments is evidently risky , and building a high-fidelity simulator can be costly . Therefore a key step towards more practical RL algorithms is the ability to learn from static data . Such paradigm , termed ” offline RL ” or ” batch RL ” , would enable better generalization by incorporating diverse prior experience . Moreover , by leveraging and reusing previously collected data , off-policy algorithms such as SAC ( Haarnoja et al. , 2018 ) has been shown to achieve far better sample efficiency than on-policy methods . The same applies to offline RL algorithms since they are by nature off-policy . The aforementioned design principles motivated a surge of recent works on offline/batch RL ( Fujimoto et al. , 2019 ; Kumar et al. , 2019 ; Wu et al. , 2019 ; Siegel et al. , 2020 ) . These papers propose remedies by regularizing the learner to stay close to the logged transitions of the training datasets , namely the behavior policy , in order to mitigate the effect of bootstrapping error ( Kumar et al. , 2019 ) , where evaluation errors of out-of-distribution state-action pairs are never corrected and hence easily diverge due to inability to collect new data samples for feedback . There exist claims that offline RL ∗Correspondence to : Lanqing Li < lanqingli1993 @ gmail.com > , Dijun Luo < dijunluo @ tencent.com > †Work done while an intern at Tencent AI Lab . 1Source code : https : //github.com/FOCAL-ICLR/FOCAL-ICLR/ can be implemented successfully without explicit correction for distribution mismatch given sufficiently large and diverse training data ( Agarwal et al. , 2020 ) . However , we find such assumption unrealistic in many practices , including our experiments . In this paper , to tackle the out-of-distribution problem in offline RL in general , we adopt the proposal of behavior regularization by Wu et al . ( 2019 ) . For practical RL , besides the ability to learn without exploration , it ’ s also ideal to have an algorithm that can generalize to various scenarios . To solve real-world challenges in multi-task setting , such as treating different diseases , driving under various road conditions or growing diverse crops in autonomous greenhouses , a robust agent is expected to quickly transfer and adapt to unseen tasks , especially when the tasks share common structures . Meta-learning methods ( Vilalta & Drissi , 2002 ; Thrun & Pratt , 2012 ) address this problem by learning an inductive bias from experience collected across a distribution of tasks , which can be naturally extended to the context of reinforcement learning . Under the umbrella of this so-called meta-RL , almost all current methods require on-policy data during either both meta-training and testing phases ( Wang et al. , 2016 ; Duan et al. , 2016 ; Finn et al. , 2017 ) or at least testing stage ( Rakelly et al. , 2019 ) for adaptation . An efficient and robust method which incorporates both fully-offline learning and meta-learning in RL , despite few attempts ( Li et al. , 2019b ; Dorfman & Tamar , 2020 ) , has not been fully developed and validated . In this paper , under the first principle of maximizing practicality of RL algorithm , we propose an efficient method that integrates task inference with RL algorithms in a fully-offline fashion . Our fully-offline context-based actor-critic meta-RL algorithm , or FOCAL , achieves excellent sample efficiency and fast adaptation with limited logged experience , on a range of deterministic continuous control meta-environments . The primary contribution of this work is designing the first end-to-end and model-free offline meta-RL algorithm which is computationally efficient and effective without any prior knowledge of task identity or reward/dynamics . To achieve efficient task inference , we propose an inverse-power loss for effective learning and clustering of task latent variables , in analogy to coulomb potential in electromagnetism , which is also unseen in previous work . We also shed light on the specific design choices customized for OMRL problem by theoretical and empirical analyses . 2 RELATED WORK . Meta-RL Our work FOCAL builds upon the meta-learning framework in the context of reinforcement learning . Among all paradigms of meta-RL , this paper is most related to the context-based and metric-based approaches . Context-based meta-RL employs models with memory such as recurrent ( Duan et al. , 2016 ; Wang et al. , 2016 ; Fakoor et al. , 2019 ) , recursive ( Mishra et al. , 2017 ) or probabilistic ( Rakelly et al. , 2019 ) structures to achieve fast adaptation by aggregating experience into a latent representation on which the policy is conditioned . The design of the context usually leverages the temporal or Markov properties of RL problems . Metric-based meta-RL focuses on learning effective task representations to facilitate task inference and conditioned control policies , by employing techniques such as distance metric learning ( Yang & Jin , 2006 ) . Koch et al . ( 2015 ) proposed the first metric-based meta-algorithm for few-shot learning , in which a Siamese network ( Chopra et al. , 2005 ) is trained with triplet loss to compare the similarity between a query and supports in the embedding space . Many metric-based meta-RL algorithms extend these works ( Snell et al. , 2017 ; Sung et al. , 2018 ; Li et al. , 2019a ) . Among all aforementioned meta-learning approaches , this paper is most related to the contextbased PEARL algorithm ( Rakelly et al. , 2019 ) and metric-based prototypical networks ( Snell et al. , 2017 ) . PEARL achieves SOTA performance for off-policy meta-RL by introducing a probabilistic permutation-invariant context encoder , along with a design which disentangles task inference and control by different sampling strategies . However , it requires exploration during meta-testing . The prototypical networks employ similar design of context encoder as well as an Euclidean distance metric on deterministic embedding space , but tackles meta-learning of classification tasks with squared distance loss as opposed to the inverse-power loss in FOCAL for the more complex OMRL problem . Offline/Batch RL To address the bootstrapping error ( Kumar et al. , 2019 ) problem of offline RL , this paper adopts behavior regularization directly from Wu et al . ( 2019 ) , which provides a relatively unified framework of several recent offline or off-policy RL methods ( Haarnoja et al. , 2018 ; Fujimoto et al. , 2019 ; Kumar et al. , 2019 ) . It incorporates a divergence function between distributions over state-actions in the actor-critic objectives . As with SAC ( Haarnoja et al. , 2018 ) , one limitation of the algorithm is its sensitivity to reward scale and regularization strength . In our experiments , we indeed observed wide spread of optimal hyper-parameters across different meta-RL environments , shown in Table 4 . Offline Meta-RL To the best of our knowledge , despite attracting more and more attention , the offline meta-RL problem is still understudied . We are aware of a few papers that tackle the same problem from different angles ( Li et al. , 2019b ; Dorfman & Tamar , 2020 ) . Li et al . ( 2019b ) focuses on a specific scenario where biased datasets make the task inference module prone to overfit the state-action distributions , ignoring the reward/dynamics information . This so-called MDP ambiguity problem occurs when datasets of different tasks do not have significant overlap in their stateaction visitation frequencies , and is exacerbated by sparse rewards . Their method MBML requires training of offline BCQ ( Fujimoto et al. , 2019 ) and reward/dynamics models for each task , which are computationally demanding , whereas our method is end-to-end and model-free . Dorfman & Tamar ( 2020 ) on the other hand , formulate the OMRL as a Bayesian RL ( Ghavamzadeh et al. , 2016 ) problem and employs a probabilistic approach for Bayes-optimal exploration . Therefore we consider their methodology tangential to ours . 3 PRELIMINARIES . 3.1 NOTATIONS AND PROBLEM STATEMENT . We consider fully-observed Markov Decision Process ( MDP ) ( Puterman , 2014 ) in deterministic environments such as MuJoCo ( Todorov et al. , 2012 ) . An MDP can be modeled as M = ( S , A , P , R , ρ0 , γ ) with state space S , action space A , transition function P ( s′|s , a ) , bounded reward function R ( s , a ) , initial state distribution ρ0 ( s ) and discount factor γ ∈ ( 0 , 1 ) . The goal is to find a policy π ( a|s ) to maximize the cumulative discounted reward starting from any state . We introduce the notion of multi-step state marginal of policy π as µtπ ( s ) , which denotes the distribution over state space after rolling out π for t steps starting from state s. The notation Rπ ( s ) denotes the expected reward at state s when following policy π : Rπ ( s ) = Ea∼π [ R ( s , a ) ] . The state-value function ( a.k.a . value function ) and action-value function ( a.k.a Q-function ) are therefore Vπ ( s ) = ∞∑ t=0 γtEst∼µtπ ( s ) [ R ( st ) ] ( 1 ) Qπ ( s , a ) = R ( s , a ) + γEs′∼P ( s′|s , a ) [ Vπ ( s′ ) ] ( 2 ) Q-learning algorithms are implemented by iterating the Bellman optimality operator B , defined as : ( BQ̂ ) ( s , a ) : = R ( s , a ) + γEP ( s′|s , a ) [ max a′ Q̂ ( s′ , a′ ) ] ( 3 ) When the state space is large/continuous , Q̂ is used as a hypothesis from the set of function approximators ( e.g . neural networks ) . In the offline context of this work , given a distribution of tasks p ( T ) where every task is an MDP , we study off-policy meta-learning from collections of static datasets of transitions Di = { ( si , t , ai , t , s′i , t , ri , t ) |t = 1 , ... , N } generated by a set of behavior policies { βi ( a|s ) } associated with each task index i . A key underlying assumption of meta-learning is that the tasks share some common structures . By definition of MDP , in this paper we restrict our attention to tasks with shared state and action space , but differ in transition and reward functions . We define the meta-optimization objective as L ( θ ) = ETi∼p ( T ) [ LTi ( θ ) ] ( 4 ) where LTi ( θ ) is the objective evaluated on transition samples drawn from task Ti . A common choice of p ( T ) is the uniform distribution on the set of given tasks { Ti|i = 1 , ... , n } . In this case , the meta-training procedure turns into minimizing the average losses across all training tasks θ̂meta = arg min θ 1 n n∑ k=1 E [ Lk ( θ ) ] ( 5 )

Summary: This paper tackles the problem of offline meta reinforcement learning, where an agent aims to learn a policy which can adapt to an unseen task (dynamics/reward), but from entirely offline data. As a result of being fully offline, the agent can no longer explore in the new task at test time, but instead receives randomly sampled transitions from the new task, from which it must infer the task. They then propose a method for learning task inference from fully offline data, as well as a policy conditioned on this task encoding from offline data built off behavior regularized actor critic (BRAC). Results indicate that in this outperforms PEARL as well as multi-task offline RL with BCQ.