Unsupervised Meta-Learning for Reinforcement Learning

1 INTRODUCTION . Reusing past experience for faster learning of new tasks is a key challenge for machine learning . Meta-learning methods achieve this by using past experience to explicitly optimize for rapid adaptation ( Mishra et al. , 2017 ; Snell et al. , 2017 ; Schmidhuber , 1987 ; Finn et al. , 2017a ; Gupta et al. , 2018 ; Wang et al. , 2016 ; Al-Shedivat et al. , 2017 ) . In the context of reinforcement learning ( RL ) , meta-reinforcement learning ( meta-RL ) algorithms can learn to solve new RL tasks more quickly through experience on past tasks ( Duan et al. , 2016b ; Gupta et al. , 2018 ; Finn et al. , 2017a ) . Typical meta-RL algorithms assume the ability to sample from a pre-specified task distribution , and these algorithms learn to solve new tasks drawn from this distribution very quickly . However , specifying a task distribution is tedious and requires a significant amount of supervision ( Finn et al. , 2017b ; Duan et al. , 2016b ) that may be difficult to provide for large , real-world problem settings . The performance of meta-learning algorithms critically depends on the meta-training task distribution , and meta-learning algorithms generalize best to new tasks which are drawn from the same distribution as the meta-training tasks ( Finn & Levine , 2018 ) . In effect , meta-RL offloads much of the design burden from algorithm design to designing a sufficiently broad and relevant distribution of meta-training tasks . While this offloading helps in acquiring representations for fast adaptation to the specified task distribution , specifying this is often tedious and challenging . A natural question is whether we can do away with manual task design and develop meta-RL algorithms that learn only from unsupervised environment interaction . In this paper , we take an initial step toward the formalization and design of such methods . Our goal is to automate the meta-training process by removing the need for hand-designed metatraining tasks . To that end , we introduce unsupervised meta-RL : meta-learning from a task distribution that is acquired automatically , rather than requiring manual design of the meta-training tasks . Unsupervised meta-RL methods must solve two difficult problems together : meta-RL with broad task distributions , and unsupervised exploration for proposing a wide variety of tasks for meta-learning . Since the assumptions of our method differ fundamentally from prior meta-RL methods ( we do not assume access to hand-specified meta-training tasks that use human-specified reward functions ) , the best points of comparison for our approach are learning meta-test tasks entirely from scratch with conventional RL algorithms . Our method can also be thought of as automatically acquiring an environment-specific learning procedure for deep neural network policies , somewhat related to data-driven initialization procedures explored in supervised learning ( Krähenbühl et al. , 2015 ; Hsu et al. , 2018 ) . The primary contributions of our work are to propose a framework for unsupervised meta-RL ; to sketch out a family of unsupervised meta-RL algorithms ; to provide a theoretical derivation that allows us to reason about the optimality of unsupervised meta-RL methods in terms of mutual information objectives ; and to describe an instantiation of an algorithm from this family that builds on a recently proposed procedure for unsupervised exploration ( Eysenbach et al. , 2018 ) and modelagnostic meta-learning ( MAML ) ( Finn et al. , 2017a ) . In addition to our theoretical derivations , we provide an empirical evaluation that studies the performance of two variants of our approach on simulated control tasks . Our experimental evaluation shows that , for a variety of tasks , unsupervised meta-RL can effectively acquire RL procedures that perform significantly better than standard RL methods that learn from scratch , without requiring additional task knowledge . 2 RELATED WORK . Our work lies at the intersection of meta-RL , goal generation , and unsupervised exploration . Metalearning algorithms use data from multiple tasks to learn how to learn , acquiring rapid adaptation procedures from experience ( Schmidhuber , 1987 ; Naik & Mammone , 1992 ; Thrun & Pratt , 1998 ; Bengio et al. , 1992 ; Hochreiter et al. , 2001 ; Santoro et al. , 2016 ; Andrychowicz et al. , 2016 ; Li & Malik , 2017 ; Ravi & Larochelle , 2017 ; Finn et al. , 2017a ; Munkhdalai & Yu , 2017 ; Snell et al. , 2017 ) . These approaches have been extended into the setting of RL ( Duan et al. , 2016b ; Wang et al. , 2016 ; Finn et al. , 2017a ; Sung et al. , 2017 ; Mishra et al. , 2017 ; Gupta et al. , 2018 ; Mendonca et al. , 2019 ; Houthooft et al. , 2018 ; Stadie et al. , 2018 ; Rakelly et al. , 2019 ; Nagabandi et al. , 2018a ; Houthooft et al. , 2018 ) . In practice , the performance of meta-learning algorithms depends on the user-specified meta-training task distribution . We aim to lift this limitation and provide a general recipe for avoiding manual task engineering for meta-RL . To that end , we make use of unsupervised task proposals . These proposals can be obtained in a variety of ways , including adversarial goal generation ( Sukhbaatar et al. , 2017 ; Held et al. , 2017 ) , information-theoretic methods ( Gregor et al. , 2016 ; Eysenbach et al. , 2018 ; Co-Reyes et al. , 2018 ; Achiam et al. , 2018 ) , and even random functions . We argue that , theoretically , methods based on mutual information have the potential to provide optimal task proposals for unsupervised meta-RL . Exploration methods that seek out novel states are also closely related to goal generation methods ( Pathak et al. , 2017 ; Schmidhuber , 2009 ; Bellemare et al. , 2016 ; Osband et al. , 2016 ; Stadie et al. , 2015 ) , but do not by themselves aim to generate new tasks or learn to adapt more quickly to new tasks , only to achieve wide coverage of the state space . These methods are complementary to our approach , but address a distinct problem . While model-based RL methods ( Chua et al. , 2018 ; Srinivas et al. , 2018 ; Deisenroth & Rasmussen , 2011 ; Nagabandi et al. , 2018b ; Atkeson & Santamaria , 1997 ) might likewise use unsupervised experience to learn a dynamics model , they are not equipped with a strategy to employ that model to learn effectively at test time . Related to our work are prior methods that study the training of goal-conditioned policies ( Schaul et al. , 2015 ; Pong et al. , 2018 ; Andrychowicz et al. , 2017 ) . Indeed , our theoretical derivation first studies the goal reaching case , before generalizing it to the more general case of arbitrary tasks . This generalization allows unsupervised meta-learning methods to solve arbitrary tasks at meta-test time without being restricted to a task parameterization or goal specification . Additionally , in Section 3.4 we discuss why the framework of meta-learning gives us theoretical benefits over the goal reaching paradigm . 3 UNSUPERVISED META-REINFORCEMENT LEARNING . In this paper , we consider the problem of automatically tailoring ( i.e. , learning ) a reinforcement learning algorithm to solve tasks on a specific environment . This learning algorithm should be meta-learned without requiring any human supervision or specification of task distribution . The implicit assumption in this formulation will be that all test-time tasks share the same dynamics but use different reward functions . To talk about environments on which many tasks might be performed , we consider a controlled Markov process ( CMP ) – a Markov decision process without a reward function , C = ( S , A , P , γ , ρ ) , with state space S , action space A , transition dynamics P , discount factor γ and initial state distribution ρ . The CMP along with this reward function rz produces a Markov decision processes Mi = ( S , A , P , γ , ρ , rz ) . The goal of the learning algorithm f is to learn an optimal policy π∗i ( a | s ) for any reward function rz that is provided with the CMP . With this notation in place , we formally define the unsupervised meta-RL as follows : Definition 1 . Unsupervised Meta-Reinforcement Learning : Given a CMP C , output a learning algorithm f that can learn new tasks efficiently in MDPs defined by CMP C together with an ( unknown ) reward function rz . The data available to unsupervised meta-RL are unsupervised interactions with the CMP , and does not include observing any of the reward functions rz . We illustrate this problem setting in Figure 1 . In contrast , in standard meta-reinforcement learning , a task distribution P ( T ) is known and we simply try to optimize for a fast learning algorithm on that known task distribution . A detailed definition can be found in Section 3.2 . In the rest of this section , we will first sketch out a general recipe for an unsupervised meta-RL algorithm , present a theoretical derivation for an optimal unsupervised meta-learning method , and then instantiate a practical approximation to this theoretically-motivated approach using components from recently proposed exploration and meta-learning algorithms . 3.1 A GENERAL RECIPE . Our framework , unsupervised meta-RL , consists of a task proposal mechanism and a meta-learning method . Formally , we will define the task distribution as a mapping from a latent variable z ∼ p ( z ) to a reward function rz ( s , a ) : S ×A → R1 . That is , for each value of the random variable z , we have a different reward function rz ( s , a ) . Under this formulation , learning a task distribution amounts to optimizing a parametric form for the reward function rz ( s , a ) that maps each z ∼ p ( z ) to a different reward function . The choice of this parametric form represents an important design decision for an unsupervised meta-learning method , and the resulting set of tasks is often referred to as a task or goal proposal procedure . The second component is the meta-learning2 algorithm , which takes the family of reward functions induced by p ( z ) and rz ( s , a ) along with the associated CMP , and meta-learns a RL algorithm f that can quickly adapt to any task from the task distribution defined by 1In most cases p ( z ) is chosen to be a uniform categorical so it is not challenging to specify 2A meta-reinforcement learning algorithm learns how to learn : it uses a set of meta-training tasks to learn a learning function f , which can then learn a new task . We refer to this learned learning function f as an ” acquired reinforcement learning procedure , ” following prior work , such as MAML ( Finn et al. , 2017a ) and RL2 ( Duan et al. , 2016b ) p ( z ) and rz ( s , a ) in the given CMP . The meta-learned algorithm f can then learn new tasks quickly at meta-test time , when a user-specified reward function is actually provided . This generic design for an unsupervised meta-RL algorithm is summarized in Figure 1 . The “ no free lunch theorem ” ( Wolpert et al. , 1995 ; Whitley & Watson , 2005 ) might lead us to expect that a truly generic approach to proposing a task distribution would not yield a learning procedure f that is effective on any real tasks , or even on the meta-training tasks . However it is important to note that even without a reward function , an unsupervised meta-learning algorithm can collect and organize meaningful information about the environment . For example , this information could include policies for reaching certain states , estimates for the distances between states , a map of the environment , or a model of the dynamics . Importantly , the meta-learner is not restricted to learning information that is human-interpretable , especially since human-interpretable representations may not be optimal for fast adaptation . In the following sections , we will discuss how to formulate an optimal unsupervised meta-learner that minimizes regret on new meta-test tasks in the absence of any prior knowledge . We first discuss the notion of an optimal meta-learner and then show how we can train one without requiring task distributions to be hand-specified . A more general version of this algorithm might also use f to inform the acquisition of tasks , allowing for an alternating optimization procedure the iterates between learning rz ( s , a ) and updating f , for example by designing tasks that are difficult for the current algorithm f to handle . However , in this paper we will consider the stagewise approach , which acquires a task distribution once and meta-trains on it , leaving the iterative variant for future work .

Summary: The paper considers a particular setting of so-called meta-reinforcement learning (meta-RL) where there is a distribution over reward functions (the transition function is fixed) and with some access to this distribution, the goal is to produce a learning algorithm that  "learns well" on the distribution. At training time, a sample of reward functions are drawn and an algorithm returns a learning program that at test time, can reinforcement learn the test environment as specified by a test reward function. The paper proposes to generate a set of training reward functions instead of relying on some "manual" specification, thus the term "unsupervised" in the title. It also proposes an algorithm, basing on a recent work in skill discovery (DIAYN, not yet peer-reviewed), to find such reward functions.