Unsupervised Meta-Learning for Reinforcement Learning

1 INTRODUCTION . Reusing past experience for faster learning of new tasks is a key challenge for machine learning . Meta-learning methods achieve this by using past experience to explicitly optimize for rapid adaptation ( Mishra et al. , 2017 ; Snell et al. , 2017 ; Schmidhuber , 1987 ; Finn et al. , 2017a ; Gupta et al. , 2018 ; Wang et al. , 2016 ; Al-Shedivat et al. , 2017 ) . In the context of reinforcement learning ( RL ) , meta-reinforcement learning ( meta-RL ) algorithms can learn to solve new RL tasks more quickly through experience on past tasks ( Duan et al. , 2016b ; Gupta et al. , 2018 ; Finn et al. , 2017a ) . Typical meta-RL algorithms assume the ability to sample from a pre-specified task distribution , and these algorithms learn to solve new tasks drawn from this distribution very quickly . However , specifying a task distribution is tedious and requires a significant amount of supervision ( Finn et al. , 2017b ; Duan et al. , 2016b ) that may be difficult to provide for large , real-world problem settings . The performance of meta-learning algorithms critically depends on the meta-training task distribution , and meta-learning algorithms generalize best to new tasks which are drawn from the same distribution as the meta-training tasks ( Finn & Levine , 2018 ) . In effect , meta-RL offloads much of the design burden from algorithm design to designing a sufficiently broad and relevant distribution of meta-training tasks . While this offloading helps in acquiring representations for fast adaptation to the specified task distribution , specifying this is often tedious and challenging . A natural question is whether we can do away with manual task design and develop meta-RL algorithms that learn only from unsupervised environment interaction . In this paper , we take an initial step toward the formalization and design of such methods . Our goal is to automate the meta-training process by removing the need for hand-designed metatraining tasks . To that end , we introduce unsupervised meta-RL : meta-learning from a task distribution that is acquired automatically , rather than requiring manual design of the meta-training tasks . Unsupervised meta-RL methods must solve two difficult problems together : meta-RL with broad task distributions , and unsupervised exploration for proposing a wide variety of tasks for meta-learning . Since the assumptions of our method differ fundamentally from prior meta-RL methods ( we do not assume access to hand-specified meta-training tasks that use human-specified reward functions ) , the best points of comparison for our approach are learning meta-test tasks entirely from scratch with conventional RL algorithms . Our method can also be thought of as automatically acquiring an environment-specific learning procedure for deep neural network policies , somewhat related to data-driven initialization procedures explored in supervised learning ( Krähenbühl et al. , 2015 ; Hsu et al. , 2018 ) . The primary contributions of our work are to propose a framework for unsupervised meta-RL ; to sketch out a family of unsupervised meta-RL algorithms ; to provide a theoretical derivation that allows us to reason about the optimality of unsupervised meta-RL methods in terms of mutual information objectives ; and to describe an instantiation of an algorithm from this family that builds on a recently proposed procedure for unsupervised exploration ( Eysenbach et al. , 2018 ) and modelagnostic meta-learning ( MAML ) ( Finn et al. , 2017a ) . In addition to our theoretical derivations , we provide an empirical evaluation that studies the performance of two variants of our approach on simulated control tasks . Our experimental evaluation shows that , for a variety of tasks , unsupervised meta-RL can effectively acquire RL procedures that perform significantly better than standard RL methods that learn from scratch , without requiring additional task knowledge . 2 RELATED WORK . Our work lies at the intersection of meta-RL , goal generation , and unsupervised exploration . Metalearning algorithms use data from multiple tasks to learn how to learn , acquiring rapid adaptation procedures from experience ( Schmidhuber , 1987 ; Naik & Mammone , 1992 ; Thrun & Pratt , 1998 ; Bengio et al. , 1992 ; Hochreiter et al. , 2001 ; Santoro et al. , 2016 ; Andrychowicz et al. , 2016 ; Li & Malik , 2017 ; Ravi & Larochelle , 2017 ; Finn et al. , 2017a ; Munkhdalai & Yu , 2017 ; Snell et al. , 2017 ) . These approaches have been extended into the setting of RL ( Duan et al. , 2016b ; Wang et al. , 2016 ; Finn et al. , 2017a ; Sung et al. , 2017 ; Mishra et al. , 2017 ; Gupta et al. , 2018 ; Mendonca et al. , 2019 ; Houthooft et al. , 2018 ; Stadie et al. , 2018 ; Rakelly et al. , 2019 ; Nagabandi et al. , 2018a ; Houthooft et al. , 2018 ) . In practice , the performance of meta-learning algorithms depends on the user-specified meta-training task distribution . We aim to lift this limitation and provide a general recipe for avoiding manual task engineering for meta-RL . To that end , we make use of unsupervised task proposals . These proposals can be obtained in a variety of ways , including adversarial goal generation ( Sukhbaatar et al. , 2017 ; Held et al. , 2017 ) , information-theoretic methods ( Gregor et al. , 2016 ; Eysenbach et al. , 2018 ; Co-Reyes et al. , 2018 ; Achiam et al. , 2018 ) , and even random functions . We argue that , theoretically , methods based on mutual information have the potential to provide optimal task proposals for unsupervised meta-RL . Exploration methods that seek out novel states are also closely related to goal generation methods ( Pathak et al. , 2017 ; Schmidhuber , 2009 ; Bellemare et al. , 2016 ; Osband et al. , 2016 ; Stadie et al. , 2015 ) , but do not by themselves aim to generate new tasks or learn to adapt more quickly to new tasks , only to achieve wide coverage of the state space . These methods are complementary to our approach , but address a distinct problem . While model-based RL methods ( Chua et al. , 2018 ; Srinivas et al. , 2018 ; Deisenroth & Rasmussen , 2011 ; Nagabandi et al. , 2018b ; Atkeson & Santamaria , 1997 ) might likewise use unsupervised experience to learn a dynamics model , they are not equipped with a strategy to employ that model to learn effectively at test time . Related to our work are prior methods that study the training of goal-conditioned policies ( Schaul et al. , 2015 ; Pong et al. , 2018 ; Andrychowicz et al. , 2017 ) . Indeed , our theoretical derivation first studies the goal reaching case , before generalizing it to the more general case of arbitrary tasks . This generalization allows unsupervised meta-learning methods to solve arbitrary tasks at meta-test time without being restricted to a task parameterization or goal specification . Additionally , in Section 3.4 we discuss why the framework of meta-learning gives us theoretical benefits over the goal reaching paradigm . 3 UNSUPERVISED META-REINFORCEMENT LEARNING . In this paper , we consider the problem of automatically tailoring ( i.e. , learning ) a reinforcement learning algorithm to solve tasks on a specific environment . This learning algorithm should be meta-learned without requiring any human supervision or specification of task distribution . The implicit assumption in this formulation will be that all test-time tasks share the same dynamics but use different reward functions . To talk about environments on which many tasks might be performed , we consider a controlled Markov process ( CMP ) – a Markov decision process without a reward function , C = ( S , A , P , γ , ρ ) , with state space S , action space A , transition dynamics P , discount factor γ and initial state distribution ρ . The CMP along with this reward function rz produces a Markov decision processes Mi = ( S , A , P , γ , ρ , rz ) . The goal of the learning algorithm f is to learn an optimal policy π∗i ( a | s ) for any reward function rz that is provided with the CMP . With this notation in place , we formally define the unsupervised meta-RL as follows : Definition 1 . Unsupervised Meta-Reinforcement Learning : Given a CMP C , output a learning algorithm f that can learn new tasks efficiently in MDPs defined by CMP C together with an ( unknown ) reward function rz . The data available to unsupervised meta-RL are unsupervised interactions with the CMP , and does not include observing any of the reward functions rz . We illustrate this problem setting in Figure 1 . In contrast , in standard meta-reinforcement learning , a task distribution P ( T ) is known and we simply try to optimize for a fast learning algorithm on that known task distribution . A detailed definition can be found in Section 3.2 . In the rest of this section , we will first sketch out a general recipe for an unsupervised meta-RL algorithm , present a theoretical derivation for an optimal unsupervised meta-learning method , and then instantiate a practical approximation to this theoretically-motivated approach using components from recently proposed exploration and meta-learning algorithms . 3.1 A GENERAL RECIPE . Our framework , unsupervised meta-RL , consists of a task proposal mechanism and a meta-learning method . Formally , we will define the task distribution as a mapping from a latent variable z ∼ p ( z ) to a reward function rz ( s , a ) : S ×A → R1 . That is , for each value of the random variable z , we have a different reward function rz ( s , a ) . Under this formulation , learning a task distribution amounts to optimizing a parametric form for the reward function rz ( s , a ) that maps each z ∼ p ( z ) to a different reward function . The choice of this parametric form represents an important design decision for an unsupervised meta-learning method , and the resulting set of tasks is often referred to as a task or goal proposal procedure . The second component is the meta-learning2 algorithm , which takes the family of reward functions induced by p ( z ) and rz ( s , a ) along with the associated CMP , and meta-learns a RL algorithm f that can quickly adapt to any task from the task distribution defined by 1In most cases p ( z ) is chosen to be a uniform categorical so it is not challenging to specify 2A meta-reinforcement learning algorithm learns how to learn : it uses a set of meta-training tasks to learn a learning function f , which can then learn a new task . We refer to this learned learning function f as an ” acquired reinforcement learning procedure , ” following prior work , such as MAML ( Finn et al. , 2017a ) and RL2 ( Duan et al. , 2016b ) p ( z ) and rz ( s , a ) in the given CMP . The meta-learned algorithm f can then learn new tasks quickly at meta-test time , when a user-specified reward function is actually provided . This generic design for an unsupervised meta-RL algorithm is summarized in Figure 1 . The “ no free lunch theorem ” ( Wolpert et al. , 1995 ; Whitley & Watson , 2005 ) might lead us to expect that a truly generic approach to proposing a task distribution would not yield a learning procedure f that is effective on any real tasks , or even on the meta-training tasks . However it is important to note that even without a reward function , an unsupervised meta-learning algorithm can collect and organize meaningful information about the environment . For example , this information could include policies for reaching certain states , estimates for the distances between states , a map of the environment , or a model of the dynamics . Importantly , the meta-learner is not restricted to learning information that is human-interpretable , especially since human-interpretable representations may not be optimal for fast adaptation . In the following sections , we will discuss how to formulate an optimal unsupervised meta-learner that minimizes regret on new meta-test tasks in the absence of any prior knowledge . We first discuss the notion of an optimal meta-learner and then show how we can train one without requiring task distributions to be hand-specified . A more general version of this algorithm might also use f to inform the acquisition of tasks , allowing for an alternating optimization procedure the iterates between learning rz ( s , a ) and updating f , for example by designing tasks that are difficult for the current algorithm f to handle . However , in this paper we will consider the stagewise approach , which acquires a task distribution once and meta-trains on it , leaving the iterative variant for future work .

Summary: The authors propose a framework for unsupervised meta-reinforcement learning. This aims to perform meta-learning in reinforcement learning context without an specification of the meta tasks being pre-specified. The authors propose two algorithms to acquire the task distributions (unsupervised). In particular the better performing approach relies on the recently introduced DIAYN algorithm.  Experiments are presented on several simple benchmark datasets.