Unsupervised Meta-Learning for Reinforcement Learning
Authors: Abhishek Gupta
Benjamin Eysenbach
Chelsea Finn
Sergey Levine
Venue:  (2020)
Keywords: Meta-Learning
Reinforcement Learning

Abstract: Meta-learning algorithms learn to acquire new tasks more quickly from past experience. In the context of reinforcement learning, meta-learning algorithms can acquire reinforcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks. The performance of meta-learning algorithms depends on the tasks available for meta-training: in the same way that supervised learning generalizes best to test points drawn from the same distribution as the training points, meta-learning methods generalize best to tasks from the same distribution as the meta-training tasks. In effect, meta-reinforcement learning offloads the design burden from algorithm design to task design. If we can automate the process of task design as well, we can devise a meta-learning algorithm that is truly automated. In this work, we take a step in this direction, proposing a family of unsupervised meta-learning algorithms for reinforcement learning. We motivate and describe a general recipe for unsupervised meta-reinforcement learning, and present an instantiation of this approach. Our conceptual and theoretical contributions consist of formulating the unsupervised meta-reinforcement learning problem and describing how task proposals based on mutual information can in principle be used to train optimal meta-learners. Our experimental results indicate that unsupervised meta-reinforcement learning effectively acquires accelerated reinforcement learning procedures without the need for manual task design and significantly exceeds the performance of learning from scratch.

Under review as a conference paper at ICLR 2020
UNSUPERVISED META-LEARNIE FOR REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Meta-learning algorithms use past experience to learn quickly solve new tasks. In the context of reinforcement learning, meta-learning algorithms acquire rein- forcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks. The performance of meta-learning algorithms depends on the tasks available for meta-training: the same way that supervised learning generalizes best to test points drawn from the same distribution as the training points, meta-learning methods generalize best tasks from the same distribution as the meta-trainin tasks. In effect, meta-reinforcemen learning offloads the design burden from algorithm design to task design. If we can automate the process of task devise meta-learning algorithm that is truly automated.
design well, In this work, we take a step in this direction, proposing a family of unsupervised meta-learning algorithms for reinforcement learning. We motivate and describe a general recipe for unsupervised meta-reinforcement learning, and present an instan- tiation of this approach. Our conceptual and theoretical contributions consist of formulating the unsupervised meta- rentorreiifor learning problem and describing how task proposals based on mutual information can be used to train optimal meta- learners. Our experimental results indicate that unsupervised meta-reinforcement learning effectively acquires accelerated reinforcement learning procedures without the need for manual task design and these procedures exceed the performance of learning from scratch.
INTRODUCTION
Reusing past experience for faster learning of new tasks is a key challenge for machine learning. Meta-learning methods achieve this by using past experience to explicitly optimize for rapid adap- tation (Mishra et al. 2017 Snell et al. [2017 [Schmidhuber] 1987 Finn et al. 201 Gupta et al.)
2018 Wang et al. 2016 Al-Shedivat al.
2017). In the context of reinforcement learning (RL), meta-reinforcement learning (meta-RL) al algorithms can learn to solve new RL tasks more quickly through experience past tasks (Duan et al., 2016b; Gupta et al.) 2018 Finn al. 2017a). Typical meta-RL algorithms assume the ability to sample from a pre-specified task distribution, and these algorithms learn solve new tasks drawn from this distribution very quickly. However, specifying a task distribution is tedious and requires a significant amount of supervision (Finn et al.) 20176]
Duan et al.) 2016b) that may be difficult to provide for large, real-world problem settings. The performance of meta-learning algorithms critically depends on the meta-training task distribution, and meta-learning algorithms generalize best new tasks which are drawn from the same distribution as the meta-training tasks (Finn & Levine 2018). In effect, me offloads much of the design burden from algorithm design to designing sufficiently broad and relevant distribution of meta- veta-training tasks. While this offloading helps in acquiring representations for fast adaptation to the specified task distribution, specifying this is often tedious and challenging. A natural question is whether we can do away with manual task design and develop meta-RL algorithms that learn only from unsupervised environment interaction. In this paper, we take an initial step toward the formalization and design of such methods.
Our goal is to automate the meta-training process by removing the need for hand-designed meta- training tasks. To that end, we introduce unsupervised meta-RL meta-learning from task distribution that is acquired automatically, rather than requiring manual design of the meta-training tasks. Un- supervised meta-RI. methods must solve two difficult problems together: meta-RI. with broad task

---

Under review as a conference paper at ICLR 2020
distributions, and unsupervised exploration for proposing wide variety of tasks for meta-learning. Since the assumptions of our method differ fundamentally from prior meta-RL methods (we do not assume access to hand-specified meta- titaatrraiing tasks that use human-specifeed reward functions), the best points of comparison for our approach are learning meta-test tasks entirely from scratch with conventional RL algorithms. Our method can also be thought of as automatically acquiring an environment-specinn learning procedure for deep neural network policies, somewhat related to data-driven initialization procedures explored in supervised learning (Krähenbühl et al. 2015 Hsu
et al. 2018).
The primary contributions of our work are propose a framework for unsupervised meta-RL; to sketch out family of unsupervised meta-RL algorithmms; to provide a theoretical derivation that allows us to reason about the optimality of unsupervised meta-RI. methods terms of mutual information objectives; and to describe an instantiation of an algorithm from this family that builds on a recently proposed procedure for unsupervised exploration (Eysenbach et al., 2018) and model- agnostic meta-learning (MAML) (Finn et al.) 70) In addition to our theoretical derivations, we provide an empirical evaluation that studies the performance simulated control tasks. Our experimental evaluation shows that, for a variety of tasks, unsupervised meta-RL can effectively acquire RL procedures that perform significantly better than standard RL methods that learn from scratch, without requiring additional task knowledge.
two variants of our approach on
2
RELATED WORK
Our work lies at the intersection of meta-RL, goal generation, and unsupervised exploration. Meta- learning algorithms use data from multiple tasks to learn how to learn, acquiring rapid adaptation
procedures from experience (Schmidhuber) 1987 Naik & Mammone 1992 Thrun & Pratt 1998
Bengio et al. 1992 Hochreiter el al. 2001] Santoro et al. 2016;; Andrychowicz et al., 2016 6;
Li &
Malik
2017
Ravi & Larochelle] 2017]] Finn et al., 2017a Munkhdalai & Yu, [017]] Snell al.
2017).
These approaches have been extended into the setting of RL (Duan et al. [016bb Wang
al.
2016 Finn al. [2017a] Sung et al. 2017 Mishra et al.. 2017 Gupta et al.] [2088] | Mendonca
et al.
2019: Houthooft et al.) [018] Stadie et al. 2018]
et al. 2019 Nagabandi et al. [01] Sa
Houthooft et al. 2018). In practice, the performance of meta-learning algorithms depends on the
user-specified meta-training task distribution. We aim to lift this limitation and provide a general recipe for avoiding manual task engineering for meta-RL. To that end, we make use unsupervised task proposals. These proposals can be obtained in a variety of ways, including adversarial goal generation (Sukhbaatar et al. 2017 Held et al. 2017),
cnurmatiiontheorete methods (Gregor et al.) Eysenbach et 2018 Co- Reyes al. 2018:] Achiam et al. (018)) and even random functions. We argue that, theoretically. methods based on mutual information have the potential to provide optimal task proposals for unsupervised meta-RL.
Exploration methods that seek out novel states are also closely related to goal generation meth- ods (Pathak et al.) 2017 Schhiidhuber] 2009 Bellemare et al. 2016 Osband et al. 2016 Stadie et al.) 2015), but do not by themselves aim to generate new tasks or learn to adapt more quickly to new tasks, only achieve wide coverage of the state space. These methods are complementary to our approach, but address a distinct problem. While based RL methods (Chua et al. [2018] Srinivas al. 2018 Deisenroth Rasmussen 2011 Nagabandi et al.) 2018b] Atkeson & Santamaria, 1997) might likewise use unsupervised experience to learn dynamics model, they are not equipped with strategy to employ that model to learn effectively at test time.
Related to our work are prior methods that study the training of goal-conditioned policies (Schaul ct al., 2015; Pong et al. 2018 Andrychowiez et al. 2017). Indeed, our theoretical derivation first studies the goal reaching case, before generalizing it to the more general case of arbitrary tasks. This generalization allows unsupervised meta-learning methods to solve arbitrary tasks at meta-test time without being restricted to task parameterization goal specification. Additionally, in Section 3.4 we discuss why the framework of meta-learming gives us theoretical benefits over the goal reaching paradigm.
2

---

Under review as a conference paper at ICLR 2020
Fast Adaptation
Urrepervieed Acquaitinn
Meta
Meta-learned
reward-maximizing policy
environment
environment-spedda RL algorithm
Unsupervised Mela-RL
reward function
Unsupervised meta-reinforcemen learning: Given an environment, unsupervised meta- RL produces an environment-speen sppecifc learning algorithm that quickly acquire new policies that maximize any task reward function.
Figure 1:
3 UNSUPERVISED REETEEEVEEEEEEEAEREEEERRCEEEEEEE LEARNING
In this paper, we consider the problem of automatically tailoring (i.e., learning) reinforcement learning algorithm to solve tasks on a specific environment. This learning algorithm should be meta-learned without requiring any human supervision specification of task distribution. The implicit assumption in this formulation will be that all test-time tasks share the same dynamics but use different reward functions.
To talk about environments on which many tasks might be performed, we consider a controlled Markov process (CMP) Markov decision process without reward function, C' (S, A, p. P), with state space S, action space A, transition dynamics P. discount factor y and initial state distribution p. The CMP along with this reward function r= produces a Markov decision processes M; = (S, A, P. P. rz). The goal of the learning algorithm f is to learn an optimal policy nī (a. s) for any reward function Tz that is provided with the CMP. With this notation in place, we formally define the unsupervised meta-RL follows:
Definition 1. Unsupervised Meta-Reinforcemen Learning: Given a CMP C, output learning algorithm f that can learn new tasks efficiently in MDPs defined by CMP C together with an (unknown) reward function rz.
The data available to unsupervised meta-RL. are unsupervised interactions with the CMP, and does not include observing any of the reward functions r.- We illustrate this problem setting in Figure [ In contrast, in standard meta-reinffrremmett learning, task distribution P(T") is known and we simply try to optimize for a fast learning algorithm on that known task distribution. detailed definition can be found in Section 3.2
In the rest of this section, we will first sketch out a general recipe for an unsupervised meta-RL algorithm, present theoretical derivation for an optimal unsupervised meta-learning method, and then instantiate a practical approximation to this theoretically approach using components from recently proposed exploration and meta-learning algorithms.
3.1 GENERAL RECIPE
Our framework, unsupervised meta-RI., consists of a task proposal mechanism and a meta-learning method. Formally, we will define the task distribution as mapping from latent variable p(z) to a reward function (s, a): x A I That is, for each value of the random variable z, we have different reward function (s, a). Under this formulation, learning a task distribution amounts to optimizing a parametric form for the reward function r (8, a) that maps each 2 ~ p(2) to a different reward function. The choice of this parametric form represents an unsupervised meta-learning method, and the resulting set of tasks is often referred to as task goal proposal procedure. The second component is the meta-learning algorithm, which takes the family of reward functions induced by p(z) and r. (8, a) along with the associated CMP, and meta-learns RL algorithm ff that can quickly adapt to any task from the task distribution defined by
important design decision for
is not challenging specify 2 A rentorreiifrr learning algorithm leamss how to learn: uses set of meta- aett-training tasks to leam a learning function which can then learn new task. We refer to this learned learning function f as an "acquired reinforcement learning procedure," following prior work, such MAMI. (Finn et al. (01 and RI.2 (Duan) 2016b)
In most cases p(z) is chosen be uniform categorical

---

Under review as a conference paper at ICLR 2020
p(z) and (= a) in the given CMP. The meta-leamed algorithm can then learn new tasks quickly at meta-test time, when a user-specified reward function is actually provided. This generic design for an unsupervised meta-RL algorithm is summarized in Figure
The "no free lunch theorem" (Wolpert et al., 1995] Whitley & Watson 2005) might lead us to expect that a truly generic approach to proposing task distribution would not yield learning procedure f that is effective on any real tasks, or even on the meta-training tasks. However is important to note that even without a reward function, an unsupervised meta-learning algorithm can collect and organize meaningful information about the environment. For example, this information could include policies for reaching certain states, estimates for the distances between states, a map of the environment, or model of the dynamics. Importantly, the meta-learner is not restricted to learning information that human-interpen numan-interpretable. especially since numan-interpretaan not be optimal for fast adaptation. In the following sections, we will discuss how to formulate an optimal unsupervised meta-learne that minimizes regret on new meta-test tasks in the absence of any prior knowledge. We first discuss the notion of optimal meta-learner and then show how we can train one without requiring task distributions to hand-specified.
representations may
A more general version of this algorithm might also use ʃ to inform the acquisition of tasks, allowing for an alternating optimization procedure the iterates between learning (s a) and updating f. for example by designing tasks that are difficult for the current algorithm to handle. However, in this paper we will consider the stagewise approach, which acquires a task distribution once and meta-trains on it, leaving the iterative variant for future work.
3.2 OPTIMAL META-LEARNERS
We first define an abstract notion of an optimal meta-learner, when given task distribution. This definition will be useful showing that for certain test-time task distributions, meta-training on self-proposee tasks (i.e., unsupervised meta-training) will yield the optimal meta-leamer at test-time. We assume that an optimal meta-learner takes in distribution over tasks for given CMP and outputs a learning procedure f that minimizes expected regret when learning tasks drawn from the same distribution as seen in meta-training. As before, the task distribution is defined by a variable z ~ p(=) and reward function r= (s, a) : x A - IR. An optimal meta- meta-leamnr optimizes
min REGRET p(r:)) = min Resptz) [R(J", r=) R(J, r=)],
where p(r;) is a distribution over reward functions, parameterized by z, RRf, r2) is the total return obtained by using meta learner ff on reward function rs, and fr E arg max E.p [R([R(""rr)) is an optimal meta-learner. This is equivalent to the expected reward objective used by most RL methods (Finn et al. 2017a| Duan et al. 2016b). Note that the behavior of the optimal meta-learner depends the specific task distribution. For example, the optimal behavior for manipulation tasks involves moving a robot's ' arms, while the optimal behavior for locomotion tasks involves moving a robot's ' legs. Optimizing Equation requires inferring task. While lcarning an internal model of the environment might assist in these steps, an optimal meta-learner would also need to know how to leverage this model. In the remainder, we will abstract away in the internal representation of the meta-learner and instead focus on the effect of the training task distribution on the behavior induced in an optimal meta-learner, and how this suggests method to construct unsupervised meta-learning algorithms.
tasks and acquiring the policy for each
3.3 SPECIAL CASE: GOAL-REACHING TASKS
We will now derive an optimal unsupervised meta-learner for the special case of goal reaching tasks, and then generalize this approach to solve arbitrary tasks in Section [34 In both, we will restrict our attention to CMPs with deterministic dynamics.
this goal-reaching setting, we consider episodes with finite horizon T and a discount factor of y 1. Tasks correspond to reaching an unknown goal state sg. We will only consider the agent's 's state at the last time step in each episode, so the (unknown) reward function is always of the form
(st) A 1(t= 1(: = g))
We will first assume that goal states are drawn from some known distribution p(sg), and later will show how remove this assumption. We define of (s) the probability that policy visits
4

---

Under review as a conference paper at ICLR 2020
state time step = T. sy is the true goal, then the event that the policy reaches 8y at the final step of an episode is Bernoulli random variable with parameter p p (sy). Thus, the expected hitting time of this goal state is
HITTINGTIME,(5)) P (5gg
We now define meta-leamer fa in terms of an exploration policy л. Before the goal is found, f uses policy и to explore. Once the goal Sg is found, the meta-learner takes actions to always return to state 8g. Meta-learner fz incurs one unit of regret for each step before it has found the goal, and zero regret afterwards. The expected cumulative regret is therefore the expectation of the hitting time, taken with respect to p(sg) p(rg):
= p(5g) ds P (5g)
REGRET(fx.,PT. =
(2)
In this special case, an optimal meta-learner as defined in Section 3.2 will explore for number of episodes until it finds the goal state. After the meta-leamnr finds the goal state, it would always choose the trajectory that reaches that goal state under deterministic dynamics. Thus, the cumulative regret of the meta-learner is the number of episodes required to find the goal state. By our assumption that the meta-learner only receives information about the task if it has reached the goal state the end of the episode, the meta-learner cannot use information about multiple goals within a single episode. We can minimize the regret in Equation calculus of variations (for more details refer to Appendix C in Lee et al. (2019), the (exploration) policy for the optimal meta-learner, n." satisfies:
2 w.r.t. the marginal distribution p" Using the
Vp(sg)
(=
(3)
S Vo(s))as,
The analysis far tells us how obtain the optimal meta- if were were given the goal sampling distribution, p(8g). If we do not know this distribution, then we cannot compute the optimal policy using Equation 3 In this case, we resort to bounding the worst-case regret of our policy:
REGRETWC = min max REGRET((a.P(r.) p(sg)
Lemma 1. Let n be a policy for which p (s) is uniform. Then f+ has lowest worst-case regret.
The proof is in Appendix A Given this result, we know that the exploration policy of the optimal meta-learner should have uniform state marginal distribution. While Lemma is only directly applicable in settings where there exists a policy that achieves uniform state marginal distribution, we might expect the intuition to carry over to other settings. The minimax optimal meta-leamen corresponds to exploring over a uniform distribution over goal states, so we can acquire this meta- learner by training Constructing this distribution is hard, especially in high-dimensional state spaces. How might we propose
goal-reaching task distribution where the goals are uniformly distributed.
goals as tasks during unsupervised meta-training? The key idea is that this uniform goal proposal distribution can be obtained by maximizing the mutual information between and the final state ST:
max M(
[u(S:: 2) 2 Hyu[sT] Hyussr 2).
(5)
where y denotes probability distribution over terminal states sr and latent variables z, and P, is a subset of such distributions. Observe that this objective contains two competing terms. The first term, H² [sy], is maximized when the marginal distribution μ(sr) is uniform. The second term, Hs 2] maximized when u(ST | 2) is Dirac distribution. The distribution μ that optimizes Equation 5 is one with a uniform marginal distribution over terminal states (proof in Appendix (A):
Lemma 2. Assume there exists some distribution μ¹" E Pa such that μ²t ((T) = UNIF(S) is uniform
over goal states and jd (sT 2) d 1(sT = 82) is a Dirac distribution, centered at a state 82 specified by the latent z. Then any distribution μ that maximizes mutual information (5(3;)))
satisfies u(ST)= UNIF(S) and p(st 2) d 1(T =82)-
5

---

Under review as a conference paper at ICLR 2020
The key idea is that joint distribution jass,, 2) can be defined implicitly via a latent-conditioned policy. This policy is not a meta-learned model, but rather will become part of the task proposal mechanism. For a given prior fa(z) and latent-conditioned policy p(a | 2). the joint likelihood is [[TT=)) = ((=)p((1) P( 8t, pe)pp(a 8t; 2), and the marginal likelihood is simply given by p(5T, 2) = (TT, 2 ddsia ... aT-1- We define Pμ as all distributions μu(T,,2) that can be constructed in such way using Markovian policy. Note that the assumption in Lemma dooss not always hold, as stochasticity in the environment may mean that j((r 2) is never a Dirac distribution, or the existence of unreachable states may mean that p(ST) can never be uniform.
Finally, we construct a distribution over tasks from the joint distribution μ.. We then define our task proposal distribution by sampling ~ p(=) and using the corresponding reward function r=(ST,0T A log p(sT 2). This reward function does not depend on the action or. In the case when both the prior μu(2) and the marginal p(sr) are uniform, this reward function is equivalent to the DIAYN reward function log μ1( $T), up to additive constants. Since each latent 2 corresponds to single state Sy when the mutual information is maximized, our task distribution corresponds exactly to sampling goals uniformly when is sampled uniformly. As argued above, successfully meta-learning on the uniform goal distribution produces the minimax-optimal meta-learner"
3.4 GENERAL CASE: TRAJECTORY MATCHING TASKS
To extend the analysis in the previous section to the general case, and thereby derive a framework for optimal unsupervised meta-learning, we will consider "trajectory-matching tasks. These tasks are a trajectory-based generalization of goal reaching: while goal reaching tasks only provide a positive reward when the policy reaches the goal state, sijectory-matchingg tasks only provide a positive reward when the policy executes the optimal trajectory. The trajectory matching case is more general because, while trajectory matching can represent different goal-reaching tasks, it can also represent tasks that are not simply goal reaching, such as reaching goal while avoiding dangerous region or reaching goal particular way. As before, we will restrict our attention to CMPs with deterministic dynamics. These nor-Markovian tasks essentially amount to problem where an RL algorithm must "guess" the optimal policy, and only receives a reward its behavior is perfectly consistent with that optimal policy. We will show that mutual information between 2 and trajectories yields the minimum regret solution in this case, and then show that unsupervised meta-learning for the mrajcctory-matching task is at least as hard as unsupervised meta-learning for general tasks (though, in practice, general tasks may be easier).
Formally, we define a distribution of trajectory-matchine tasks by distribution over goal trajectories, p(7"). For each goal trajectory r", the corresponding trajectory-level reward function is
(T) A 1(T=T")
Analogous to before, we define the hitting time as the expected number of episodes to match the target trajectory:
1
HITTINGTIME, T(T*)
We then define regret as the expected hitting time:
p(T)
REGRET((z,P(r.) = HITTINGTTME;(T))()))))
dr x(T)
(8)
Using the same derivation as before, the exploration policy for the optimal meta-learner is
VP(T)
(T)
dP'
However, obtaining such a policy requires knowing the trajectory distribution p(T). In the setting where p(7) is unknown, the minimax policy is simply uniform:
IIt is important to note that the propopaoposal policy μ is not an optimal meta-learer. For example, in a k k-armed bandit, the optimal meta-learne would simply enumerate over all arms and find the best arm in k/2 steps, whereas the prok-plo policy would randomly sample arms and require approximately & log(k)/2 steps to find the best arm.

---

Under review as a conference paper at ICLR 2020
Lemma 3.
Let be policy for which T()) is uniform. Then fx has lowest worst-case regret.
How can we acquire a policy with a uniform trajectory distribution? Repeating the steps above, we learn collection of skills using a trajectory-level mutual information objective:
T(T;=) -[[7] H(T |=]
(10)
Using the same reasoning as Section 3.3 the optimal policy for this objective has a uniform distribu- tion over trajectories that, conditioned on partieular latent 2, deterministically produces a single trajectory in a deterministic CMP. Analogous to Section 3.3 we define a distribution over reward functions as (T) - log z). At optimality, each corresponds to exactly one trajectory Tz, so the reward function (=(T) simply indicates whether 7 is equal to Tz- Since the distribution over trajectories f p(= =)p(z)dz is uniform at the optimum, the distribution of reward functions rz corresponds uniform distribution over trajectories. Thus, meta-learning on the rewards from trajectory-level mutual information results in the minimax-optimal meta-learner.
Now that we have derived the optimal meta-leamer for sajeettorymmachhing tasks, observe that senjectory-matching is super-set of the problem of optimizing any possible Markovian reward function test-time. For a given initial state distribution, each reward function is optimized by a particular trajectory. However, trajectories produced by non-Markovian policy (i.e., policy with memory) are not necessarily the unique optimum for any Markovian reward function. Let Rt denote the set of trajectory-Ieve reward functions, and Re.o denote the set of all state-action level reward functions. Bounding the worst-case regret on R, minimizes an upper bound on the worst-case regret on Rg.a.
[E
min E., [r+(T]] S min Ex ERE
for all policies π.
reR
This inequality holds for all policies n, including the policy that maximizes the LHS. While we aim to maximize the RHS, we only know how to maximize the LHS. which gives lower bound on the RHS. This incquality holds for all policies т, so it also holds for the policy that maximizes the LHS. In general, this bound is loose because the set of all Markovian reward functions is smaller than the set of all trajectory-level reward functions (i.e., maatchhn tasks). However, this bound becomes tight when considering meta-learning on the set of all possible (non-Markovian) reward functions.
In the discussion of meta-learning thus far, we have considered tasks where the reward is provided at the last time step T of each episode. In this particular case, the best that an optimal meta-learner can do is go directly goal or execute a particular trajectory at every episode according to the p((s.)
vp(r)
optimal exploration policy: (T- = V for goal reaching or x² (T() = J p for
trajectory matching. All intermediate states in the trajectory are uninformative, thus making instances of meta-learning algorithms which explore via schemes like posterior sampling optimal for this class of problems. In the more general case with arbitrary reward functions, intermediate rewards along trajectory may be informative, and the optimal exploration strategy may be different from posterior sampling (Rothfuss et al., 2019 Duan al. 2016b; Wang et al. 2016). Nonetheless, the analysis presented in this section provides us insight into the behavior of optimal meta-learning algorithms and allows us to understand the qualities desirable for unsupervised task proposals. The general proposed scheme for unsupervised meta-learning has a number of benefits over standard universal value function and goal reaching style algorithms:
(1) it can be applied to arbitrary reward functions going beyond simple goal reaching and (2) it can acquire a significantly more effective exploration strategy than posterior sampling by using intermediate states a trajectory for intelligent information gathering.
3.5 SUMMARY OF ANALYSIS
Through our amalysis, we introduced the notion of optimal meta-learners and analyze their exploration behavior and regret on a class of goal reaching problems. We showed that on these problems, when the test-time task distribution is unknown, the optimal meta-training task distribution for minimizing worst-case test-time regret is uniform over the space of goals. We also showed that this optimal task distribution can be acquired by a simple mutual information maximization scheme. We subsequently extend the analysis to the more general case of matching arbitrary trajectories, proxy for the
7

---

Under review as a conference paper at ICLR 2020
more general class of arbitrary reward functions. In the following section, will discuss how can derive a practical algorithm for unsupervised neta-learning from this analysis.
3.6 A PRACTICAL ALGORITHM
Following the derivation in the previous section, instantiate practical unsupervised meta- RL algorithm by constructing a task proposal mechanism based on a mutual information objec- tive. A variety of different mutual information objectives can be formulated, including mutual information between single states and x (Eysenbach et al.] 2018), pairs of start and end states and (Greorr et al., 2016), and entire trajectories and (Achiam al., 2018). We will use DIAYN and leave a full examination of possible mutual information objectives for future work.
DIAYN optimizes mutual information by train- ing discriminator network Da(=l.) that pre- diets which z was used to generate the states in
Algorithm 1: Unsupervised Meta-Reinforcemeet
Learning Pseudocode
given rollout according to latent-conditioned policy т T(a 2). Our task proposal distribution thus defined by (s, a) = log( D (z]s)) al-
Data: M \ R. an MDP without a reward function
Result: learning algorithm f: Dg +
DR DIAYN() D' random while not converged do
The complete unsupervised meta-learning gorithm follows the recipe in Figure [ first, we acquire (2(88 a) by running DIAYN, which learns (2(2(s) and a latent-conditionee policy π [alss, z) (which is discarded). Then, we use p(z) to propose tasks (s, a) a standard
Sample latent task variables x ~ p(z)
Extract corresponding task reward functions (s())
using Ds(a(8)
Update using MAML with reward (s)
meta-RL algorithm. This meta-RL algorithm uses the proposed tasks to learn how to learn, acquiring fast learn algorithm f which can then learn new tasks quickly. While, in principle, any meta-RL. RL algorithm could be used, the discussion at the end of Section 3.4 suggests that meta-RL methods based on posterior sampling might suboptimal for some tasks. We use MAML (Finn et al.) 2017a) meta- meta-learning algorithm. Note that the learning algorithm f returned by MAML is defined simply as running gradient descent using the parameters found by MAML as initialization. While not immediately apparent how parameter initialization define entire learning algorithm, insightful prior work Finn & Levine (0177) provides an in-depth discussion of this property. The complete method is summarized in A Algorithm|
In addition to mutual information maximizing task proposals, we will also consider random task proposals, where also discriminator the reward, according to r(s, 2) log Derum (=(s). but where the parameters grand are chosen randomly (i.e., a random weight initialization for a neural network). While such random reward functions are not optimal, we find that they can be used to acquire useful task distributions for simple tasks, though they are not as effective as the tasks become more complicated. This empirically reinforces the claim that unsupervised meta-RI. does not fact violate any "no free lunch" principle even simple task proposals that cause the meta-learner to explore the CMP can already accelerate learning of new tasks.
EXPERIMENTAL EVALUATION
In our experiments, we aim to understand whether unsupervised meta-learning as de- scribed in Section can can provide us with an ac- celerated RL procedure on new tasks. Whereas standard meta-learning requires hand-specified task distribution at meta-training time, unsuper- vised meta- meta-learning learns the task distribution
Half-Cheetah
Ant
through unsupervised interaction with the environment. A fair baseline that likewise uses requires no reward supervision at training time, and only uses rewards at test time, is learning via RL from scratch without any meta-leant leta-learning. As an upper bound, we include the unfair comparison to standard meta-learning approach, where the meta-training distribution is manually designed. This method has access to a hand-specifier task distribution that is not available to our method. We evaluate two variants of our approach: (a) task acquisition based DIAYN followed by meta-learning using
8

---

Under review as a conference paper at ICLR 2020
192
2D navigation
Half-Cheetah
Ant
Figure 3: Unsupervised meta-learning accelerates learning: After unsupervised meta-lcaming. our approach (UML-DLAYN and UML-RANDOM) quickly learns a new task significantly faster than learning from scratch, especially on complex tasks. Learning the task distribution with DIAYN helps more for complex tasks. Results are averaged across 20 evaluation tasks, and random seeds for testing. UML-DIAYN and random also significantly outperform learning with DIAYN initialization or an initialization with a policy pretrained with VIME.
MAML, and (b) task acquisition using randomly initialized discriminator followed by meta-learning using MAML.
4.1 TASKS AND IMPLEMENTATION DETAILS
Our experiments study three simulated environments of varying difficulty: 2D point navigation, 2D locomotion using the "HalfCheetah," and 3D locomotion using the "Ant," with the latter two environments are modifications of popular RL benchmarks (Duan et al. 2016a). While the 2D navigation environment allows for direct control of position, HalfCheetah and Ant can only control their center of mass via feedback control with high dimensional actions (6D for HalfCheetah, 8D for Ant) and observations (17D for HalfCheetah, 111D for Ant).
The evaluation tasks, shown in Figure 6 are similar to prior work (Finn et al. 2017a Pong et al. 2018): 2D navigation and ant require navigating to goal positions, while the half cheetah must run different goal velocities. These tasks are not accessible to our algorithm during meta-training, We used the default hyperparameters for MAML across all tasks, varying the meta-batch size according to the number of skills: 50 for pointmass, 20 for cheetah, and 20 ant. We found that the default architecture, 2 layer MLP with 300 units each and ReLU non-lineasisies, worked quite well for meta- meta-training training. We also used the default hyperparameters for DIAYN to acquire skills. We swept over learning rates for learning from scratch via vanilla policy gradient, and found that using ADAM with adaptive step size is the most stable and quick at learning.
4.2 FAST ADAPTATION AFTER UNSUPERVISED META LEARNING
The comparison between the two variants of unsupervised meta-learning and learning from scratch is shown in Figure 3 We also add a comparison to VIME (Houthooft et al., 2016), a standard novelty- based exploration method, where pretrain policy with the VIME reward and then finetune on the meta-test tasks. In all cases, the UML-DIAYN variant of unsupervised meta-learning produces an RL procedure that outperforms RL from scratch and VIME-init, suggesting that unsupervised interaction with the environment and meta-learning is effective in producing environment-specinn but task-agnostic priors that accelerate learning on new, previously unseen tasks. The comparison with VIME shows that the speed of learning is not just about exploration but is indeed about fast adaptation. In our experiments thus far, UML-DIAYN always performs better than learning from scratch, although the benefit varies across tasks depending on the actual performance of DIAYN. We added additional comparisons to using TRPO as a stronger optimizer from scratch with the same trends being observed. We also compared with baseline of simply initializing from DIAYN trained contextual policy, and then finetuning the best skill with the actual task reward. From the comparisons in Fig it is apparent that this works poorly compared with UMRL variants.
Interestingly, in many cases (in 4 the performance of unsupervised meta-learning with DIAYN matches that of the hand-designed task distribution. We see that on the 2D navigation task, while handcrafted meta- leeanle is able to learn very quickly initially, it performs similarly after 100 steps. For the cheetah environment well, handcrafted meta-learning is able to learn very quickly to start off, but is quickly matched by unsupervised meta-RL with DIAYN. On the ant task, we see that hand-crafted meta-learning does do better than UML-DIAYN, likely because the task distribution is

---

Under review as a conference paper at ICLR 2020
Navigation
CheetCh
Ant
tonk hardle (00/Se
Scrrdh
Nonkested
Handoaattd
2D Navigation
Half-Chcetah
Ant Navigation
Figure 4: Comparison with handcrafted tasks: Unsupervised meta- meta-learning LUML-DIAYN is competitive with meta-training on handcrafted reward functions (i.e., an oracle). A misspecified. handcrafted meta-training task distribution often performs worse, illustrating the benefits learning task distribution.
more challenging, and a better unsupervised task proposal algorithm would improve the performance of meta-learner.
The comparison between the two unsupervised meta-learning variants is also illuminating: while the DIAYN-based variant of our method generally achieves the best performance, even the random discriminator is often able to provide a sufficient diversity of tasks to produce meaningful acceleration over learning from scratch in the case of 2D navigation and ant. This result has two interesting implications. First, suggests that unsupervised meta-learning an effective tool for learning an environment prior. Although the performance of unsupervised meta-learning can be improved with better coverage using DIAYN (as seen in Figure 3). even the random discriminator version provides competitive advantages over learning from scratch. Second, the comparison provides clue for identifying the source of the structure learned through unsupervised meta-learning: though the particular task distribution has effect on performance, simply interacting with the environment (without structured objectives, using random discriminator) already allows meta-RL to learn effective adaptation strategies given environment. That is, the performance cannot be explained only by the unsupervised procedure (DIAYN) capturing the right task distribution. We also provide an analysis of the task distributions acquired by the DIAYN procedure in Appendix [.I B.1
DISCUSSION AND FUTURE WORK
We presented an unsupervised approach to meta-RL, where meta-learning is used to acquire an efficient RL procedure without requiring hand-specified task distributions for meta-training. This approach accelerates RL without relying on the manual supervision required for conventional meta- learning algorithms. We provide a theoretical derivation that argues that task proposals based mutual information maximization can provide for minimum worst-case regret meta-learner, under certain assumptions. We then instantiate an approximation to the theoretically coooetically-mooivateedd motivated method by building on recently developed unsupervised task proposal and meta-learning algorithms. Our experiments indicate that unsupervised meta-RL can accelerate learning range of tasks, outperforming learning from scratch and often matching the performance meta-learning from hand-specified task distributions.
As our work is the first foray into unsupervised meta-RL, our approach opens a number of questions about unsupervised meta-leaming algorithms. One limitation of our analysis is that it only considers deterministic dynamics, and only considers task distributions where posterior sampling is optimal. Extending analysis to stochastic dynamics and more realistic task distributions may allow unsupervised meta-RL to acquire learning algorithms that can explore and adapt more intelligently, and more effectively solve real-world tasks.
REFERENCES
Joshua Achiam, Harrison Edwards. Dario Amodei. and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv:1807, 10299, 2018.
Maruan Al- Al-Shedivat, Trapit Bansal, Yuri Burda, 1lya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-lcarning in nonstationary and competitive environments. arXiv preprint arxi:: 1710.03611, 2017.
10

---

Under review as a conference paper at ICLR 2020
Marcin Andrychowicz, Misha Denil, Sergio Gomez. Matthew W Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. Neural Information Processing Systems (NIPS), 2016.
Marcin Andrychowiez. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048-5058, 2017.
Christopher G Atkeson and Juan Carlos Santamaria. A comparison of direct and model- based reinforcement learning. In Proceedings of International Conference on Robotics and Automation, volume 4, pp. IEFE, 1997.
Mare G. Bellemare, Sriram Srinivasan, Georg Ostrovski. Tom Schaul, David Saxton, and Rémi Munos. Unifying boset exploration and intrinsic motivation. CoRR, abs//606.01 2016. URL hctp:://///ax
org/abs/ /166 01868
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of synaptic learning rule. In Optimality in Artificial and Biological Neural Nenworks, 1992.
Kurtland Chua, Roberto Calandra, Rowan McAllister and Sergey Levine. Deep reinforcement learning in handful trials using probabilistic dynamics models. Advances Neural Information Processing Systems, pp. 4754-4765, 2018.
John D Co- Reyes, YuXuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey Levine. Self-consistent trajectory autoencoder: Hicrarchical reinforcement learning with trajectory embeddings. arXiv preprint arXiv:/806. 1...0//3006028/3, 2018.
Mare Deisenroth and Carl E. Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICMI--I), pp. 465-472, 2011.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Picter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp.
2016a.
Yan Duan, John Schulman, Xi Chea, Peter L Bartlett Ilya Sutskever, and Pieter Abbeel. RI Fast reinforcement learning via slow reinforcement learning, arXiv preprint orXiv:1611.02775 2016b.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint 18Xtv:8802.00770 2018.
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any lcarning algorithm. CoRR, 11s/1700.11622,, 2017. URLhttp://arxi org/ abs/
11602.1662
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. International Conference Learning Representations, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnosiie meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.034000 2017a.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. CoRR, (bs//179904495. 2017b. ULL//ttpp://xxxxv.oo////b/
C4905
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv.1611.07507. 2016.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta-reinforcement learning of structured exploration strategies. arXiv preprint 1rXiv:1802.00725,, 2018.
David Held, Xinyang Geng, Carlos Florensa, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. arXiv preprint 17Xx:11705.06666 2017.
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, 2001.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Picter Abbecl. VIME: variational information maximizing exploration. In Advances in Neural Information Processing Systems, 2016.
Rein Houthooft, Richard Y Chen, Phillip Isola, Bradly C Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. arXiv preprint arxiv: 288820042// 2018

---

Under review as a conference paper at ICLR 2020
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. arXiv preprint arXiv:1810002334, 2018.
Philipp Krähenbühl, Carl Doersch, Jeff Donahue, and Trevor Darrell. Data-dependent initializations of convolu- tional neural networks. arXiv preprint
2015.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric P. Xing. Sergey Levine, and Ruslan Salakhutdinov.
Efficient exploration via state marginal matching. CoRR, abs/1906.05274, 2019. URL/btp:://axxy
org/abs/ aaaaabss9900 /1906 05274
Ke Li and Jitendra Malik. Learning to optimize. International Conference Learning Representations (ICLR), 2017.
Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta-policy search. CoRR, (bss/190400956, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- meta-learner. In NIPS 20/7 Workshop Meta-Learring, 2017.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. International Conference on Machine Learning (ICML), 2017.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel. Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-rcinforccmment learning. arXiv preprint 18X3 1347, 2018 8a.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model- based deep reinforcement learning with model-free fine-tuning, In 2018 IEEE International Conference on Robotics and Automation (ICRA). pp. 7559-7566. IEEE, 2018b.
Devang K Naik and RJ Mammone. Meta-neural networks that learn by learning. In International Joint Conference on Neural Netowrks (IJCNN), 1992.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped 04621
DQN. CoRR, awss/6/0..04.... 2016. URL
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity- exploration by self- supervised prediction. In ICML, 2017.
Vitchyr Pong. Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model-free deep for model-based control. arXiv preprint arXiv:1802.00008 2018.
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient oof-py meta- reinforcement learning via probabilistic context variables. arXiv preprint arXivv1903.082.. 2019.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In International Conference on Learning Representations (ICLR), 2017.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abheel. Promp: Proximal meta-policy search. In International Conference on Learning Representations, ICLR, 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-Icarning with memory- augmented neural networks. In International Conference on Machine Learning (ICML), 2016.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning, pp. 1312-1320, 2015.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
Jürgen Schmidhuber. Driven by compression progress: A simple principle explains essential aspects of
subjective beauty, novelty, surprise, interestingness, attention, curiosity, crcativity, art, science, music, jokes. In Computational Creativity: An Interdisciplinary Approach, 12.07. 17.07.2009, 2009. URL
http::/ htateess.eaasaaal..ee//e/asaeete
Jake Suell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in
Neural Information Processing Systems, pp. 4080 4080-4090, 2017.
Aravind Srinivas, Allan Jabri, Picter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks. arXiv preprint arXiv:1800 804000645, 2018.
12

---

Under review as a conference paper at ICLR 2020
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:150..00814, 2015.
Bradly C. Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever.
Some considerations on learning explore rentrreeifore learning. CoRR, abs/1803.01 2018. wttp://wwwrraaaa///.aa///// /tll:tttp://aaxi..or////////////
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Syunaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self play. arXiv preprint arXiv:1703.05400 2017.
Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, and Yongxin Yang. Learning learn: Meta-critic networks for sample efficient learning. arXiv preprint arXiv: netil77669.9999 2017.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science Business Media, 1998.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, Matt Botvinick. Learning to reinforcement learn. arXiv preprint 16 2016.
Darrell Whitley Jean Paul Watson. Complexity theory and free lunch theorem. 2005.
David H Wolpert, William G Macready, al. No free lunch theorems for search. Technical report, Technical Report SF1-TR-95-02-010, Santa Fe Institute, 1995.
13

---

Under review as a conference paper at ICLR 2020
A PROOFS
Lemma Let x be policy for which (s) is uniform. Then i has lowest worst-case regret.
Proof of Lemma ! To begin, we note that all goal distributions p(sg) have equal regret for policies
where (s) S/|S| is uniform:
REGRET,(T) p(sg) dsg p (s
p(5g) 1/|5|
dsg |S|
Now, consider policy n' for which p (s) is not uniform. For simplicity, will assume that the argmin is unique, though the proof holds for non-unique argmins as well. The worst-case goal distribution will choose the state where that the policy least likely visit:
p (5g) _ 1(5 arg min pT (s))
min
Thus, the worst-case regret for policy x' is strictly greater than the regret for a uniform
max REGRET, = REGRETT (T) /
1(5g = arg min, Px (8)) dsg min, P
1
S
(s,)
(11)
Thus, policy n' for which Pf is non-uniform cannot be minimax, the optimal policy has uniform marginal p=
Lemma 2: Mutual information [ST;; 2) is maximized by task distribution p(sg) which is uniform over goal states.
Proof of Lemma We define a latent variable model, where we sample a latent variable a from a uniform prior p(2) and sample goals from a conditional distribution p(sr =). To begin, note that the mutual information can be written as a difference of entropies:
4p(3;;2) 2) HplsT] Hp[sT
The conditional entropy HplsT 2] attains the smallest possible value (zero) when each latent variable corresponds to exactly one final state, sz. contrast, the marginal entropy H,[[T]] attains the largest possible value (log (S) when the marginal distribution p(sT) = f p(sT =)p( ())2 is uniform. Thus, a task uniform distribution p(s,)) maximizes [ST: 2). Note that for any non-uniform task distribution Q(ST), we have [sT < H.lssT]. Since the conditional entropy HplST 2] is zero, no distribution can achieve a smaller conditional entropy. This, for all non-uniform task distributions q. have 2) < I (sr: 2). Thus, the optimal task distribution must be uniform.
B ABLATIONS
-100
.25
ty150
100
-40
Figure 5: Analysis of effect of additional meta-training meta-test time learning of new tasks. For larger iterations of meta-trained policies, have improved time performance, showing that additional meta- training is beneficial.
To understand the method performance more clearly, we also add an ablation study where we compare the meta-test performance of policies at different iterations along meta-training. This shows the effect
14

---

Under review as a conference paper at ICLR 2020
that additional meta- meta-training training has the fast learning performance for new tasks. This comparison is shown in Figure 5 As can be seen here, at iteration of veta-training the policy is not a very good see that
initialization for learning the meta-learned initialization becomes more and more effective at learning new tasks. This shows a clear correlation between additional meta-training and improved meta test-time performance.
tasks. As move further along the meta-training process,
B.1 ANALYSIS LEARNED TASK DISTRIBUTIONS
2D navigation
Ant
Half-Cheetah
Figure 6: Learned meta-trainde veta-training task distribution and evaluation tasks: We plot the center of mass for various skills discovered by point and using DIAYN, and blue histogram of goal velocities for cheetah. Evaluation tasks, which are not provided to algorithm during meta-training, are plotted as red pointmass, and as a green histogram for cheetah. While the meta-training distribution is broad, it does not fully cover the evaluation tasks. Nonetheless, meta-learning this learned task distribution enables efficient learning test task distribution.
ant and
We can analyze the tasks discovered through unsupervised exploration and compare them to tasks we evaluate on at meta-test time. F Gillustrates these distributions using scatter plots for 2D navigation and the Ant, and a histogram for the HalfCheetah. Note that we visualize dimensions of the state that are relevant for the evaluation tasks positions and velocities but these dimensions are not specified any way during unsupervised task acquisition, which operates on the entire state space. Although the tasks proposed via unsupervised exploration provide fairly broad coverage, they are clearly quite distinct from the meta-test tasks, suggesting the approach can tolerate considerable distributional shift. Qualitatively, many of the tasks proposed via unsupervised exploration such as jumping and falling that are not relevant for the evaluation tasks. Our choice of the evaluation tasks was largely based on prior work, and therefore for unsupervised meta-RL therefore suggest quite strongly that unsupervised task acquisition can provide an effective meta-tran titta-trannng set, least for MAML, even when evaluating on tasks that do not closely match the discovered task distribution.
tailored to this exploration procedure. The results
C HYPERPARAMETER DETAILS
For all our experiments, we used DIAYN to acquire the task proposals using 20 skills for half-cheetah and for ant and skills for the 2D navigation. We ran the domains using the standard DIAYN hyperparameters described in https:::p........... //g thub. com proposals. These proposals were then fed into the MAML algorithm https: / thub. com/ cbfinn/nii mannnnmmmmm...mmmmmmmm r1 with inner learning rate 0.1, meta learning rate 0.01, inner batch size 40, outer batch size 20, path length 100, using 2 layer networks with 300 units each with ReLu nonlinearities. The test time learning is done with the same parameters for the UMRL variants, and done using REINFORCE with the Adam optimizer for the comparison with learning from scratch.
sac to acquire task
15


Reviews:
[{"review_id": "rkgUHZwTKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper develops a meta-learning approach for improving sample efficiency of learning different tasks in the same environment. The author formulates the meta goal as minimizing the expected regret under the worst case, which happens when all the tasks are uniformly distributed. The paper introduces two types of tasks: goal-reaching task and a more general trajectory matching task. Then the author introduces a meta-learning algorithm to minimize the regret by learning the reward function under different sampled tasks. The paper is interesting. Below are my questions/concerns.\n \n1. Why trajectory matching is considered as more general? Intuitively, trajectory matching is more restricted in that whenever an agent can match the optimal trajectory, it should also reach the goal state. \n\n2. The theoretical results (lemma 2, 3) actually indicates that the previous work universal value function approximator can optimize the proposed meta learning objective with theoretical convergence guarantee in tabular case by learning the value function Q(s, g, a) where s is a state, g is goal state, a is an action (as long as s and g are visited infinitely often) . As a result, why is it necessary to introduce meta-learning approach? Why not simply learn universal value functions? \n\n3. The experimental results are not very persuasive. What is the VPG algorithm used? And if you run the algorithm longer, is it finally worse than learning from scratch? Option learning methods/universal value function can be added as baselines. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "rkeDSjvI5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary of the paper:\n\nThis paper formulates conceptually the unsupervised meta-RL problem (to learn a policy without access to any reward function) as a minimization of the expected regret over tasks, and instantiate an algorithm based on DIAYN (Eysenbach et al., 2018) and MAML Finn et al. (2017a). \n\n# Brief explanation of my rating:\n\n1. *Novelty*: Mutual information based unsupervised RL was proposed by DIAYN (Eysenbach et al., 2018). Meta-model was also considered by DIAYN (Eysenbach et al., 2018), in which they call it \"skill\". \n2. *Technical contributions*: Sec 3.1-3.4 try to justify DIAYN. However, the reasoning is not sufficiently rigorous and the proposed Algorithm 1 is inconsistent with the theory built up in these sections. \n3. The *writing* can be improved a lot -- it's not easy to guess what the author was trying to say until I read DIAYN (Eysenbach et al., 2018). \n4. The key ingredient is missing -- the learning procedure f, which was mentioned in eq.(1) and Algorithm 1, but the details are never specified. It is impossible to reproduce the algorithm based on the description in the paper. \n4. The same *experiments* are conducted in DIAYN (Eysenbach et al., 2018). I am still confused on why we suddenly should use meta-RL. \n\n# Comments:\n\n1. Why we should consider regret? What is the relation between (1) & (4)? It's quite strange you start with (1) but turn to something else, i.e., (4), quickly. \n2. \"This policy induces a distribution over terminal states, p(s_T | z)\" Why? \n3. What are you optimizing over in (5)?  The statement in Lemma 2 says \"I(s_T; z) maximized by a task distribution p(s_g)\". However, you are only able to control p(s_T | z), not the marginal distribution p(s_T). The statement of Lemma should be made more clear.\n4. The definition of the reward function: r_z(s_T, a_T) = log p(S_T | z), which is independent of the action a_T? \n5. In Algorithm 1, the reward reuse the definition of DIAYN -- log D(z | s),  but which is different from log p(S_T | z). Could you elaborate this? \n6. What is the definition of Markovian reward? Why does the inequality on page 6 hold? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lqr4oRFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: this paper claims to design an unsupervised meta-learning algorithm that does automatically design a task distribution for the target task. The conceptual idea is to propose a task based on mutual information and to train the optimal meta-learner. They also use experiments to show the effectiveness of the proposed approach. \n\nOverall Comments:\n\nI would think this paper requires a major revision. It is written in a very confusing way. Many terms are directly used without a definition. The problem is also not clearly defined. I have tried to understand everything, but I have to give up in Section 3. Overall, I do not think this paper is ready for publication.\n\nDetailed comments:\n\t• It would benefit a lot if you can clearly define the original meta-learning procedure and then compare that with the one proposed in this paper.\n\t• Define ”hand-specified” distribution. This word does not make sense if you claim this is the difference between the meta-learning procedure proposed in this paper and the original meta-learning algorithm. In this paper, you used p(z) to specify a task. I would think p(z) is also “hand-specified”.\n\t• I am not very sure by what you mean for “task-proposal procedure”, “goal-proposal procedure”\n\t• In the first paragraph of the intro: what do you mean by “specifying a task distribution is tedious”, is specifying p(z) also “tedious”\n\t• 2nd paragraph of intro: “automate the meta-training process by removing the need for hand-designed meta-training tasks”. Again, why p(z) is not “hand-designed”\n\t• Why compare with the original meta-RL algorithm on p(z) is not fair? \n\t• What do you mean by “acquire reinforcement learning procedures”?\n\t• “Environment”, “task” are not clear when they first appear\n\t• The word “learn” is used everywhere, and is confusing. E.g. what do you mean by “learn new tasks”, “learn a learning algorithm f”, “learn an optimal policy”, “learn a task distribution” …\n\t• “Reward functions induced by p(z) and r_z(s,a)”: isn’t r_z(s,a) already a reward function? What is “induced”?\n\t• What is “meta-training” time?\n\t• What is “no free lunch theorem”?\n\t• The “controlled-MDP” setting is actually much easier: perhaps you just need to learn the probability distribution. Then for every r_z, we just solve it. Why not compare with this simple algorithm?\n\t• “Regret” is not defined when it first appears\n\t• “The task distribution is defined by a latent variable z and a reward function r_z”: why “distribution” is defined by an r.v.?\n\t• In (2), “regret” should be the (cost of the algorithm) - (the total cost of an optimal policy) — it is not hitting time\n\t• (3) is confusing, no derivation is given\n\t• Based on the usual definition of “regret”, how can a “policy” have low regret? Any fixed “policy” would have linear regret …", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper discusses the relevant topic of unsupervised meta-learning in an RL setting. The topic is an interesting one, but the writing and motivation could be much clearer. I advise the authors to make a few more iterations on the paper taking into account the reviewers' comments and then resubmit to a different venue.
Decision: Reject