FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization

1 INTRODUCTION . Applications of reinforcement learning ( RL ) in real-world problems have been proven successful in many domains such as games ( Silver et al. , 2017 ; Vinyals et al. , 2019 ; Ye et al. , 2020 ) and robot control ( Johannink et al. , 2019 ) . However , the implementations so far usually rely on interactions with either real or simulated environments . In other areas like healthcare ( Gottesman et al. , 2019 ) , autonomous driving ( Shalev-Shwartz et al. , 2016 ) and controlled-environment agriculture ( Binas et al. , 2019 ) where RL shows promise conceptually or in theory , exploration in real environments is evidently risky , and building a high-fidelity simulator can be costly . Therefore a key step towards more practical RL algorithms is the ability to learn from static data . Such paradigm , termed ” offline RL ” or ” batch RL ” , would enable better generalization by incorporating diverse prior experience . Moreover , by leveraging and reusing previously collected data , off-policy algorithms such as SAC ( Haarnoja et al. , 2018 ) has been shown to achieve far better sample efficiency than on-policy methods . The same applies to offline RL algorithms since they are by nature off-policy . The aforementioned design principles motivated a surge of recent works on offline/batch RL ( Fujimoto et al. , 2019 ; Kumar et al. , 2019 ; Wu et al. , 2019 ; Siegel et al. , 2020 ) . These papers propose remedies by regularizing the learner to stay close to the logged transitions of the training datasets , namely the behavior policy , in order to mitigate the effect of bootstrapping error ( Kumar et al. , 2019 ) , where evaluation errors of out-of-distribution state-action pairs are never corrected and hence easily diverge due to inability to collect new data samples for feedback . There exist claims that offline RL ∗Correspondence to : Lanqing Li < lanqingli1993 @ gmail.com > , Dijun Luo < dijunluo @ tencent.com > †Work done while an intern at Tencent AI Lab . 1Source code : https : //github.com/FOCAL-ICLR/FOCAL-ICLR/ can be implemented successfully without explicit correction for distribution mismatch given sufficiently large and diverse training data ( Agarwal et al. , 2020 ) . However , we find such assumption unrealistic in many practices , including our experiments . In this paper , to tackle the out-of-distribution problem in offline RL in general , we adopt the proposal of behavior regularization by Wu et al . ( 2019 ) . For practical RL , besides the ability to learn without exploration , it ’ s also ideal to have an algorithm that can generalize to various scenarios . To solve real-world challenges in multi-task setting , such as treating different diseases , driving under various road conditions or growing diverse crops in autonomous greenhouses , a robust agent is expected to quickly transfer and adapt to unseen tasks , especially when the tasks share common structures . Meta-learning methods ( Vilalta & Drissi , 2002 ; Thrun & Pratt , 2012 ) address this problem by learning an inductive bias from experience collected across a distribution of tasks , which can be naturally extended to the context of reinforcement learning . Under the umbrella of this so-called meta-RL , almost all current methods require on-policy data during either both meta-training and testing phases ( Wang et al. , 2016 ; Duan et al. , 2016 ; Finn et al. , 2017 ) or at least testing stage ( Rakelly et al. , 2019 ) for adaptation . An efficient and robust method which incorporates both fully-offline learning and meta-learning in RL , despite few attempts ( Li et al. , 2019b ; Dorfman & Tamar , 2020 ) , has not been fully developed and validated . In this paper , under the first principle of maximizing practicality of RL algorithm , we propose an efficient method that integrates task inference with RL algorithms in a fully-offline fashion . Our fully-offline context-based actor-critic meta-RL algorithm , or FOCAL , achieves excellent sample efficiency and fast adaptation with limited logged experience , on a range of deterministic continuous control meta-environments . The primary contribution of this work is designing the first end-to-end and model-free offline meta-RL algorithm which is computationally efficient and effective without any prior knowledge of task identity or reward/dynamics . To achieve efficient task inference , we propose an inverse-power loss for effective learning and clustering of task latent variables , in analogy to coulomb potential in electromagnetism , which is also unseen in previous work . We also shed light on the specific design choices customized for OMRL problem by theoretical and empirical analyses . 2 RELATED WORK . Meta-RL Our work FOCAL builds upon the meta-learning framework in the context of reinforcement learning . Among all paradigms of meta-RL , this paper is most related to the context-based and metric-based approaches . Context-based meta-RL employs models with memory such as recurrent ( Duan et al. , 2016 ; Wang et al. , 2016 ; Fakoor et al. , 2019 ) , recursive ( Mishra et al. , 2017 ) or probabilistic ( Rakelly et al. , 2019 ) structures to achieve fast adaptation by aggregating experience into a latent representation on which the policy is conditioned . The design of the context usually leverages the temporal or Markov properties of RL problems . Metric-based meta-RL focuses on learning effective task representations to facilitate task inference and conditioned control policies , by employing techniques such as distance metric learning ( Yang & Jin , 2006 ) . Koch et al . ( 2015 ) proposed the first metric-based meta-algorithm for few-shot learning , in which a Siamese network ( Chopra et al. , 2005 ) is trained with triplet loss to compare the similarity between a query and supports in the embedding space . Many metric-based meta-RL algorithms extend these works ( Snell et al. , 2017 ; Sung et al. , 2018 ; Li et al. , 2019a ) . Among all aforementioned meta-learning approaches , this paper is most related to the contextbased PEARL algorithm ( Rakelly et al. , 2019 ) and metric-based prototypical networks ( Snell et al. , 2017 ) . PEARL achieves SOTA performance for off-policy meta-RL by introducing a probabilistic permutation-invariant context encoder , along with a design which disentangles task inference and control by different sampling strategies . However , it requires exploration during meta-testing . The prototypical networks employ similar design of context encoder as well as an Euclidean distance metric on deterministic embedding space , but tackles meta-learning of classification tasks with squared distance loss as opposed to the inverse-power loss in FOCAL for the more complex OMRL problem . Offline/Batch RL To address the bootstrapping error ( Kumar et al. , 2019 ) problem of offline RL , this paper adopts behavior regularization directly from Wu et al . ( 2019 ) , which provides a relatively unified framework of several recent offline or off-policy RL methods ( Haarnoja et al. , 2018 ; Fujimoto et al. , 2019 ; Kumar et al. , 2019 ) . It incorporates a divergence function between distributions over state-actions in the actor-critic objectives . As with SAC ( Haarnoja et al. , 2018 ) , one limitation of the algorithm is its sensitivity to reward scale and regularization strength . In our experiments , we indeed observed wide spread of optimal hyper-parameters across different meta-RL environments , shown in Table 4 . Offline Meta-RL To the best of our knowledge , despite attracting more and more attention , the offline meta-RL problem is still understudied . We are aware of a few papers that tackle the same problem from different angles ( Li et al. , 2019b ; Dorfman & Tamar , 2020 ) . Li et al . ( 2019b ) focuses on a specific scenario where biased datasets make the task inference module prone to overfit the state-action distributions , ignoring the reward/dynamics information . This so-called MDP ambiguity problem occurs when datasets of different tasks do not have significant overlap in their stateaction visitation frequencies , and is exacerbated by sparse rewards . Their method MBML requires training of offline BCQ ( Fujimoto et al. , 2019 ) and reward/dynamics models for each task , which are computationally demanding , whereas our method is end-to-end and model-free . Dorfman & Tamar ( 2020 ) on the other hand , formulate the OMRL as a Bayesian RL ( Ghavamzadeh et al. , 2016 ) problem and employs a probabilistic approach for Bayes-optimal exploration . Therefore we consider their methodology tangential to ours . 3 PRELIMINARIES . 3.1 NOTATIONS AND PROBLEM STATEMENT . We consider fully-observed Markov Decision Process ( MDP ) ( Puterman , 2014 ) in deterministic environments such as MuJoCo ( Todorov et al. , 2012 ) . An MDP can be modeled as M = ( S , A , P , R , ρ0 , γ ) with state space S , action space A , transition function P ( s′|s , a ) , bounded reward function R ( s , a ) , initial state distribution ρ0 ( s ) and discount factor γ ∈ ( 0 , 1 ) . The goal is to find a policy π ( a|s ) to maximize the cumulative discounted reward starting from any state . We introduce the notion of multi-step state marginal of policy π as µtπ ( s ) , which denotes the distribution over state space after rolling out π for t steps starting from state s. The notation Rπ ( s ) denotes the expected reward at state s when following policy π : Rπ ( s ) = Ea∼π [ R ( s , a ) ] . The state-value function ( a.k.a . value function ) and action-value function ( a.k.a Q-function ) are therefore Vπ ( s ) = ∞∑ t=0 γtEst∼µtπ ( s ) [ R ( st ) ] ( 1 ) Qπ ( s , a ) = R ( s , a ) + γEs′∼P ( s′|s , a ) [ Vπ ( s′ ) ] ( 2 ) Q-learning algorithms are implemented by iterating the Bellman optimality operator B , defined as : ( BQ̂ ) ( s , a ) : = R ( s , a ) + γEP ( s′|s , a ) [ max a′ Q̂ ( s′ , a′ ) ] ( 3 ) When the state space is large/continuous , Q̂ is used as a hypothesis from the set of function approximators ( e.g . neural networks ) . In the offline context of this work , given a distribution of tasks p ( T ) where every task is an MDP , we study off-policy meta-learning from collections of static datasets of transitions Di = { ( si , t , ai , t , s′i , t , ri , t ) |t = 1 , ... , N } generated by a set of behavior policies { βi ( a|s ) } associated with each task index i . A key underlying assumption of meta-learning is that the tasks share some common structures . By definition of MDP , in this paper we restrict our attention to tasks with shared state and action space , but differ in transition and reward functions . We define the meta-optimization objective as L ( θ ) = ETi∼p ( T ) [ LTi ( θ ) ] ( 4 ) where LTi ( θ ) is the objective evaluated on transition samples drawn from task Ti . A common choice of p ( T ) is the uniform distribution on the set of given tasks { Ti|i = 1 , ... , n } . In this case , the meta-training procedure turns into minimizing the average losses across all training tasks θ̂meta = arg min θ 1 n n∑ k=1 E [ Lk ( θ ) ] ( 5 )

Summary: The paper studies meta-reinforcement learning in the fully offline setting, and proposes a novel algorithm 'FOCAL'. Given offline datasets for tasks sampled from some prior, the algorithm learns a context encoder using distance-based metrics. The encoder is used for inferring the task-latent $z$, which is used to condition the policy rollout $\pi(a | s, z)$. They demonstrate experiments where FOCAL outperforms baselines like PEARL.