Understanding the Complexity Gains of Contextual Multi-task RL with Curricula
Authors: Qiyang Li
Yuexiang Zhai
Yi Ma
Sergey Levine
Venue: Submitted to ICLR 2023 (2023)
Keywords: policy gradient methods
multi-task RL

Abstract: Reinforcement learning (RL) problems can be challenging without well-shaped rewards. Prior work on provably efficient RL methods generally proposes to address this issue with dedicated exploration strategies, such as novelty-based bonuses. However, another way to tackle this challenge is to reformulate it as a multi-task RL problem, where the task space contains not only the challenging task of interest but also easier tasks that implicitly function as a curriculum. Such a reformulation opens up the possibility of running existing multi-task RL methods as a more efficient alternative to solving a single challenging task from scratch. In this work, we provide a theoretical framework that reformulates a single-task RL problem as a multi-task RL problem defined by a curriculum. Under mild regularity conditions on the curriculum, we show that sequentially solving each task in the multi-task RL problem is more computationally efficient than solving the original single-task problem, without any explicit exploration bonuses or other exploration strategies. We also show that our theoretical insights can be translated into an effective practical learning algorithm that can accelerate curriculum learning on simulated robotic goal-reaching tasks.

Under review as a conference paper at ICLR 2023
UNDERSTANDING THE COMPLEXITY GAINS OF REFOR- MULATING SINGLE-TASK RL WITH A CURRICULUM
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning (RL) problems can be challenging without well-shaped rewards. Prior work provably efficient RL methods generally proposes address this issue with dedicated exploration strategies. However, another way to tackle this challenge is reformulate it as multi-task RL problem, where the task space contains not only the challenging task of interest but also easier tasks that implicitly function as a curriculum. Such reformulation opens up the possibility of running existing multi-task RL methods as more efficient alternative to solving a single challenging task from scratch. In this work, we provide a theoretical framework that reformulates single-task RL problem defined by a curriculum. Under mild regularity conditions on the curriculum, we show that sequentially solving each task in the multi-task RL problem is more computationally efficient than solving the original single-task problem, without any explicit exploration bonuses other exploration strategies. We also show that our theoretical insights can be translated into an effective practical learning algorithm that can accelerate curriculum learning robotic goal-reaching tasks.
multi-task RL problem
INTRODUCTION
Reinforcement learning (RL) provides an appealing and simple way to formulate control and decision- making problems in terms of reward functions that specify what agent should do, and then automatically train policies to learn how to do it. However, in practice the specification of the reward function requires great care: if the reward function is well-shaped, then learning be fast and effective, but if rewards are delayed, sparse, or can only be achieved after extensive explorations, RL problems can be exceptionally difficult (Kakade and Langford, 2002: Andrychowicz et al., 2017; Agarwal et al., 2019). This challenge is often overcome with either reward shaping (Ng et al., 1999; Andrychowicz al., 2017: 2020; Gupta et al., 2022) or dedicated exploration methods (Tang et al., 2017; Stadie et al., 2015; Bellemare et al., 2016; Burda et al., 2018). but reward shaping can bias the solution away from optimal behavior, while even the best exploration methods, in general, may require covering the entire state space before discovering high-reward regions.
On the other hand, a number of recent works have proposed multi-task learning methods RL that involve learning contextual policies that simultaneously represent solutions tasks, such as policies that reach any potential goal (Fu et al., 2018; Eysenbach et al., 2020b; Fujita et al., 2020; Zhai et al., 2022), policies conditioned on language commands (Nair al., 2022), or even policies conditioned on the parameters of parametric reward functions (Kulkarni et al., 2016; Siriwardhana al., 2019; Eysenbach et al., 2020a; Yu al., 2020b). While such methods often not motivated directly from the standpoint of handling challenging exploration scenarios, but rather directly aim to acquire policies that can perform all tasks in the task space. these multi-task formulations often present a more tractable learning problem than acquiring a solution to a single challenging task in the task space (e.g., the hardest goal, or the most complex language command). We pose the following question: can we construct a multi-task RL problem with contextual policies that easier than solving single-task RL problem from scratch? In this work, we answer this question affirmatively by analyzing the sample complexity of class of curriculum learning methods.
entire space of
To build the intuition for how reformulating a single-task problem into a multi-task problem enables
efficient learning, consider the setting where the optimal state visitation distributions d dff of two different contexts w, w/ are "similar", and our goal is to learn the optimal policy x.. w.r.t w'. Suppose have learned the optimal policy m we can facilitate learning x by: (1) using m as

---

Under review as a conference paper at ICLR 2023
an initialization and (2) setting a new initial state distribution مr' = Sd + (1 -1--88 Bpp) by mixing di the optimal state visitation distribution of Th and μu, the initial distribution of interest. Using To as initialization for learning Th facilitates the learning process as guarantees the initialization within
some neighborhood of the optimality. Setting the new initial distribution μl Bdd + (11 3) also facilitates the learning of Th as it reduces the density mismatch ratio between / / /n'1 key
quantity that influences sample complexity. The new distribution μ' Bd + (1 8) for learning The could be understood as "rolling in di with probability 8". We refer to this approach, which consists of rolling in similar optimal state visitation distribution of another context, ROLLIN.
We illustrate the intuition of ROLLIN in Figure 1. More specifically, we adopt the contextual MDP formulation, where we assume each MDP M.,, is uniquely defined by a context w in the context space W R", and we given a curriculum {wx KK with the last MDP Maxx being the MDP of interest. To show our main results, we only require a Lipschitz continuity assumption on Tw w.r.t. w and some mild regularity conditions on the curriculum {wx K We show that learning i by recursively rolling in with a near- optimal policy for wis to construct the initial distribution M4+1 for the next context W is provably more efficient than learning Tex from scratch. In particular, we show that when an appropriate sequence of contexts is selected, can reduce the iteration and sample complexity bounds of heroyy-rgglariiii softmax policy gradient (with an inex- act stochastic estimation of the gradient) from an original exponential dependency on the state space size, sug- gested by Ding et al. (2021), to polynomial dependency. We also prescribe a practical implementation of ROLLIN.
p((n))
Learning mhe from scratch
Learning Th with ROLLIN
Figure I Illustration of ROLLIN. The red circle represents initial state distribution. The dark curve represents the optimal pol- icy w.r.t. w. The blue diamonds represent the optimal state distributions d dfff'' of w, w' respectively. For learning Th when the optimal policy Th of similar context is known, ROLLIN rolls in the optimal policy of near by context w with probability B.
In summary, our contributions can be stated as follows. We first provide theoretical method (ROLLin) that facilitates single-task policy learning by recasting it as multi-task problem under cntropy-regularizee complexity bound of the cntrropy-regularizee PG to polynomial dependency on S. Last but not least, we also provide a deep RL implementation of ROLLIN and demonstrate adding ROLLIN improves performance in simulated goal-reaching tasks with an oracle curriculum and non-oracle curriculum learned from MEGA (Pitis et al., 2020), as well as several standard Mujoco locomotion tasks inspired by meta RL (Clavera et al., 2018).
softmax policy gradient (PG), which reduces the exponential
2 RELATED WORK
Convergence of policy gradient methods. Theoretical analysis of policy gradient methods has a long history (Williams, 1992; Sutton et al., 1999; Konda and Tsitsiklis, 1999; Kakade and Langford, 2002; Peters and Schaal, 2008). Motivated by the recent empirical success (Schulman et al., 2015;
2017) in policy gradient (PG) methods, the theory community has extensively studied the convergence of PG in various settings (Fazel et al., 2018; Agarwal et al., 2021; 2020; Bhandari and Russo, 2019;
Mei et al., 2020; Zhang et al., 2020b; Agarwal et al., 2020; Zhang al., 2020a; Li et al., 2021;
Cen et al., 2021; Ding et al., 2021; Yuan et al., 2022; Moskovitz et al., 2022). Agarwal et al. (2021) established the asymptotic global convergence of policy gradient under different policy parameterizations. We extend the result of entropy regularized PG with stochastic gradient (Ding et al., 2021) to the contextual MDP setting. In particular, our contextual MDP setting reduces the exponential state space dependency w.r.t. the iteration number and per iteration sample complexity suggested by Ding et al. (2021) a polynomial dependency. We shall also clarify that there much existing convergence analysis on other variants of PG that produce an iteration number that does not suffer from an exponential state space dependency (Agarwal et al., 2021; Mei et al., 2020), but they assume access to the exact gradient during each update of PG, while we assume a stochastic estimation of the gradient, which is arguably more practical.
Contextual MDPs Contextual MDPs (or MDPs with side information) have been studied exten- sively in the theoretical RI. literature (Abbasi- Yadkori and Neu, 2014; Hallak et al., 2015; Dann et al.,
2

---

Under review as a conference paper at ICLR 2023
2019; Jiang et al., 2017; Modi et al., 2018; Sun et al., 2019; Dann et al., 2019; Modi al., 2020). We analyze the iteration complexity and sample complexity of (stochastic) policy gradient methods, which is distinct from these prior works that mainly focus regret bounds (Abbasi- Yadkori and Neu, 2014; Hallak et al., 2015; Dann et al., 2019) and PAC bounds (Jiang et al., 2017; 7; Modi et al., 2018; Sun al., 2019; Dann et al., 2019; Modi et al., 2020). Several works assumed linear transition kernel and reward model (or generalized linear model (Abbasi- Abbasi-Yadkorr Yadkori and Neu, 2014)) with respect to the context (Abbasi- Yadaki and Neu, 2014; Modi et al., 2018; Dann et al., 2019; Modi al., 2020; Belogolovsky et al., 2021). These assumptions share similarity to our assumptions we have a weaker Lipschitz continuity assumption with respect to the context space (since linear implies Lipschitz) on the reward function and stronger shared transition kernel assumption.
Exploration. A number of prior works have shown that one can reduce the complexity of learning an optimal policy with effective exploration methods (Azar et al., 2017; Jin al., 2018; Du et al., 2019; Misra et al., 2020; Agarwal et al., 2020; Zhang et al., 2020d). The computational efficiency suggested by our work is different from some of the aforementioned prior methods that rely on adding exploration bonus for policy cover (Azar et al.. 2017: Jin et al., 2018; Agarwal et al., 2020; Zhang et al., 2020d), as we assume access "good" curriculum which ensures the optimal policy defined by the next context is not too different from the optimal policy of the current context. The key intuition of our work is more related to the strategic exploration in the block MDP setting (Du et al., 2019; Misra et al., 2020). The block MDP setting assumes the state space can be encoded into a feature space with a sequential block structure and explores the next block in the feature space after having a policy that covers the previous blocks. Such sequential exploration procedure is similar to ROLLIN, as our method also considers solving the multi- multi-task contextual MDPs in a sequential perspective, but ROLLIN focuses learning the optimal policy of the next context once we have near-optimal policy of the current context.
Curriculum learning in reinforcement learning. Curriculum learning is a powerful idea that has been widely used in RL (Florensa et al., 2017; Kim and Choi, 201 8; Omidshafiei et al., 2019; Ivanovic et al., 2019; Akkaya et al., 2019; Portelas et al., 2020; Bassich et al., 2020; Fang et al., 2020; Klink et al., 2020; Dennis et al., 2020; Parker-Holder et al., 2022; Liu et al., 2022) (also see (Narvekar et al., 2020) for a detailed survey). Although curricula formed by well-designed reward functions ( (Vinyals et al., 2019; OpenAl, 2018; Berner et al., 2019; Ye et al., 2020; Zhai et al., 2022) are usually sufficient given enough domain knowledge, tackling problems with limited domain knowledge requires a more general approach where a suitable curriculum is automatically formed from task space. In the goal-conditioned reinforcement learning literature, this corresponds to automatic goal proposal mechanisms (Florensa et al., 2018; Warde-Farley al., 201 8; Sukhbaatar et al., 2018; Ren et al., 2019; Ecoffet et al.. 2019; Hartikainen et al., 2019; Pitis et al.. 2020; Zhang et al., 2020c; OpenAI al., 2021: Zhang et al., 2021). The practical instantiation of this work is also similar Bassich et al. (2020); Liu et al. (2022), where a curriculum is adopted for learning a progression of a set of tasks.
Learning conditional policies in multi-task RL. Multi-task RL (Tanaka and Yamamura, 2003) approaches usually learn a task-conditioned policy that is shared across different tasks (Rusu et al., 2015; Rajeswaran et al., 2016; Andreas et al., 2017: Finn et al., 2017: D'Eramo et al., 2020; Yu et al., 2020a; Ghosh et al., 2021; Kalashnikov et al., 2021). Compared to learning each task independently, joint training enjoys the sample efficiency benefits from sharing the learned experience across different tasks as long as the policies generalize well across tasks. To encourage generalization, is often desirable to condition policies on low dimensional feature representations that are shared across different tasks instead (c.g.. using variational antoo-encoders (Nair al., 2018; Pong et al., 2019; Nair et al., 2020) or variational information bottleneck (Goyal et al., 2019; 2020; Mendonca et al., 2021)). The idea of learning contextual policies has also been discussed in classical adaptive control literature (Sastry et al., 1990; Tao, 2003; Landau et al., 2011; Áström and Wittemmark, 2013; Goodwin and Sin. 2014). Different from these prior works which have been mostly focusing on learning policies that can generalize across different tasks, our work focuses on how the netrmoptimal policy from a learned task could be used to help the learning of similar task.
3 PRELIMINARIES
We consider the contextual MDP setting, where contextual MDP Mw SW Tw. 7,p)
consists of a context space W. a state space S, an action space A, a transition dynamic function
P. S A- P(S) (where P(X) denotes the set of all probability distributions over set X).
a context-conene conditioned reward function r : W x S x A + [o, 1]. a discount factor y E (0, and an initial state distribution of interest p. For convenience, use S [8], A |A] to
3

---

Under review as a conference paper at ICLR 2023
denote the number of states and actions. While some contextual MDP formulations (Hallak et al., 2015) have context-conditione transition dynamics and reward functions, we consider the setting where only the reward function can change across contexts. We denote Tu as the reward function conditioned on a fixed w E W and MM = (S,A, Pur as the MDP in- duced by such fixed reward function. We use ((ass) : S P(A) denote policy and adopt the softmax parameterization:
TT(aas) = E., exp["(=,a']]' where 0 S x A R. We use (s) := (1 7) Ei (P s S/80 ~ p) to denote the discounted state visitation distribu- tion and V = 1o (s,, at)] + a m) to denote the entropy regularized discounted return on Mu. where H((p,)) == [oopp)a 1hh=o -7" log π [an]ss)) is the discounted entropy term. We use Th == arg max, V to denote an optimal policy that maximizes the dis- counted return under M We assume all the contextual reward functions are bounded within [0. 1]: (s, E 10, 1), Vo E Ω v(s E S x A. Similarly to previous analysis Agarwal et al. (2021); I); Mei et al. (2020); Ding et al. ), we assume the initial distribution for PG or stochastic PG satisfies p(s) > 0, VS S.
exp]
3.1 ASSUMPTIONS
Given curriculum [wx]/=o,, where the last context wk defines Maxe the MDP of interest, our goal show that reformulating Max into multi-task problem {Mur eejoeooss better computational complexity and sample complexity than solving the single-task problem .Mux from scratch. As we will show in Section 4, if we have a "good" curriculum {wx } such that the optimal policies Twy Tenys w.r.t two consecutive contexts wk; W++1 are "close enough" to each other, then using an e-optimal policy of w/ as an initialization allows us to directly start from the near optimal regime of Wk+1, hence only requiring polynomial complexity to learn al Formally, we use the two following assumptions to characterize the good properties of our curriculum.
Assumption 3.1 (Lipschitz reward in the context space) The reward function is Lipschitz contin- uous w.r.f the context: max,,a (s, a) Tw. a)l S L, || w'll2- Yw, E W.
Assumption 3.2 (Similarity between two Consecutive Contexts) We assume the given curricu- lum {w>}k=o satisfies maxosk<K-1 || - Wall2 S O (S-2). and we have access to a near- optimal initialization 0 for learning Two (formally define in Section 4.2).
Intuitively, Assumption 3.1 defines the similarity between two tasks via a Lipschitz continuity the context space, similar Lipschitz assumption also appears in Abbasi- et al. (2018); Dann et al. (2019); Modi et al. (2020); Belogolovsky et al. (2021). Assumption 3.1 and 3.2 together quantify the maximum difference between two consecutive tasks M M in terms of the maximum difference between their reward function. Assumption 3.1 and 3.2 also play a crucial role in reducing the exponential complexity to a polynomial one, and will briefly discuss the intuition in the next section.
and Neu (2014); Modi
3.2 PRIOR ANALYSIS ON PG WITH STOCHASTIC GRADIENT
Ding et al. (2021) proposed a two-phased PG convergence analysis framework with a stochastic gradient. In particular, the author demonstrates that with high probability, stochastic PG with arbitrary
initialization achieves an e-optimal policy can be achieved with an iteration number of Tı, T2 and per
iteration sample complexity of B1, B2 in two separate phases where Tı = (2(52 ). T2 (2533/2) (2() suppresses the log S and terms that do not contain S) and B (2(525) ). Bg = (($5)) respectively, and PG enters phase 2 only when the updating policy becomes En cn-optimal, where Eo is a term depending on (formally defined by (19) in Appendix A.3). For completeness, we restate the main theorem of Ding al. (2021) in Theorem A.2, provide the details of such dependencies on S are provided in Corollary A.3, and describes the two phase procedure in Algorithm 4. The main implication of these two-phase results is that, if we want to apply PG to learn optimal policy from an arbitrary initialization, we suffer from an exponential iteration number and sample complexity, unless the initialization is co-optimal. In the next section, will discuss how Assumption 3.1 and 3.2 enables an éo opttmaal initialization for every We hence reducing the exponential complexity to a polynomial one.
4 THEORETICAL ANALYSIS
In this section, we introduce ROLLIN, a simple algorithm that accelerates policy learning under the contextual MDP setup by bootstrapping new context learning with better initial distribution (Algorithm 1). We also provide the total complexity analysis of applying ROLLIN to stochastic PG for achieving an s-optimal policy.
4

---

Under review as a conference paper at ICLR 2023
4.1 ROLLIN
The theoretical version of ROLLIN is provided in Algorithm 1. that when two consecutive contexts in the curriculum (wx) are close, their optimal parameters
Our intuition behind ROLLIN is
8% : 8 should be close to each other. Let 0(*) denote the parameters at the tt iteration of stochastic PG for learning 02y- if we initialize the parameter 0 the optimal parameter of the previous context 0- Wk-1 (line in Algorithm 1), and set the initial distribution P4 as a mixture between
-in- and the original distribution of the optimal state visitation distribution of the previous context dpa-1+ interest p with ratio (line 6 in Algorithm 1), such that
μk 3dpx ----- +(1 - (1-B)p,
(1)
then stochastic PG enjoys faster convergence rate. This happens because setting 0(4) -
ensures near-optimal initialization for 0(*), and setting Mh by mixing dyi-."" and p further improves dhx the rate of convergence by decreasing the density mismatch ratio | /pL 1ill term with an S dependency that influences the rate of convergence).
Algorithm Provably Efficient Learning via ROLLIN
1: Input: P. {ax Ik Mw, (0, 1), 1).66
2: Initialize (uu p.
3: Run stochastic PG (Algorithm 4) with initialization ( 14, Man and obtain 0
4: for k = :==...., K do
5: Set 0 = 8 W
Tww- moi,-1 is the optimal policy of w b Pyp is the initial distribution for learning 0i,-- 7: Run stochastic PG (Algorithm 4) with initialization 0(*), uk, Muz and obtain We"
6: Set L4 = 3dpx
+ 101-399 (pp
8: end for
9: Output: 8%
4.2 MAIN RESULTS
We now discuss how use sequence of contexts to learn the target context wk with provable efficiency given the optimal policy ni, of the initial context wi, without incurring an exponential dependency S (as mentioned in Section 3.2). Our polynomial complexity comes as result of enforcing an ontommall initialization (En is the same as Section 3.2 and (19)) for running stochastic PG (line 6 of Algorithm 1). Hence, stochastic PG directly enters phase 2, with polynomial dependency in S.
Our main results consist of two parts. We first show that when consecutive contexts Wk wk-1,46 Wk are close enough to each other, using ROLLIN for learning 0% with initialization g(*) = 0% and
applying an initial distribution μıx = " + (1 - p improves the convergence rate. Specifically, the iteration number and complexity for learning 0% from 02 is stated follows: Theorem 4.1 (Complexity of Learning the Next Context) Consider the context-based stochastic softmax policy gradient (line 7 of Algorithm 1), suppose Assumption 3.1 and Assumption 3.2 hold, then the iteration number of obtaining an e-optimal policy for wi from wi-1 is St (5322) and the
per iteration sample complexity is Φi a(1--))
In other words, Theorem 4.1 shows that when Wk-1, We are close enough, ROLLIN reduces the iteration and sample complexity from exponential dependency of ))(525)) to an iteration number of (3/22) and per iteration sample complexity of 52(53). is worth noting that the theorem above only address the iteration numbers and sample complexity for learning 6%, from 6%,
Theorem 4.3 provides the total complexity for learning 0:x from 0²") via recursively apply the results in Theorem 4.1. Before introducing Theorem 4.3, we first provide a criteria for the desired initialization of (())
Definition 4.2 (Near-optimal Initialization) We say Og is a near-optimal initialization for learning 0% if fg satisfies - (p) V.550 (p) < E0-
5

---

Under review as a conference paper at ICLR 2023
Note that in the above definition, The represents the optimal policy of wk. and V represents value function of context Wy under policy x. Now we introduce the results for the overall complexity:
Theorem 4.3 (Main Results: Total Complexity of ROlLIN) Suppose Assumption 3.1 and 3.2
hold, and 0%) is an near-optimal initialization, then the total number of iteration of learning Tux using Algorithm/ is (3/2)) and the per iteration is St (S*), with high probability.
A direct implication of Theorem 4.3 is that, with a curriculum {w k=0 satisfying Assumption 3.1 and 3.2, one can reduce the daunting exponential dependency on S caused by poor initialization to polynomial dependency on S. Admittedly the state space S itself is still large in practice, but reducing the state space itself requires extra assumptions on S which is beyond the scope of this work. We will provide sketch proof of Theorem 4.1 and 4.3 in the next subsection and leave all the details proof to Appendix A.4, A.5 respectively.
1K
4.3 PROOF SKETCH
Sketch proof of Theorem 4.1 The key insight for proving Theorem 4.1 is to show that in MDP Ma the value function w.r.t. The, Twa-1 can be bounded by the e² norm between wk and Wk-1- In particular, we prove such a relation Lemma A.5:
(p) 2L, 1w/k Wk-1ll
(p)
Vwk
(2)
wk
71-
By setting 0(*) = 0 Equation (2) directly implies V (0) - v (p) S
2T, Bao M wy
ille
(1
As suggested by Ding al. (2021) stochastic PG directly start from stage 2 with polynomial
complexity of T2 (2(3332), B₂ (((S),, if V (p) v0 (p) S Eg. where En (formally defined
in Equation (19) in Appendix A.3) is a constant satisfying 6o = ((-22). Hence, by enforcing two consecutive contexts to be close enough ||4 - wa-illl 2I, O(S-2), )) can directly start from near-optimal initialization with polynomial complexity w.r.t. S. This largely explains the intuition behind the initialization in line 5 of ROLLIN: 0(() 6% 1' It is worth highlighting that the per iteration sample complexity B2 shown by Ding al. (2021) scales $2(55), while our result in Theorem 4.1 only requires smaller sample complexity ((S*). Such an improvement in the [d + (1 B)p. Intuitively, setting
sample complexity comes from line 6 of ROLLIN: as (d + (1 P) allows us to provide an upper bound on the density mismatch ratio:
J /uK
< (
a(1 4.t).
L,
(3)
where A maxigisk Iw; wi-1 l. Since the sample complexity B3 (provided in Corollary A.3)
contains one multiplier of d/t 1 setting A = 0(-22) immediately reduces the complexity
by an order of S?. The proof of the upper bound of the density mismatch ratio (Equation 3) is provided in Lemma A.
Sketch proof of Theorem 4.3 We obtain Theorem 4.3 by recursively applying Theorem 4.1. More precisely, we use induction to show that, we initialize the parameters of the policy as 8(*) 0m,
when t = ((S3322, 4k E [K]. we have Vu THi (p) - Vaz Tath (p) < En- Hence, for any context
Wk, A {K], initializing 8(4) = 0(t from learning T. 1 via stochastic PG after t 225$322
iteration, o(*)) will directly start from the efficient phase 2 with polynomial complexity. Hence, the total iteration number for learning the 0% is S3/23/)) and the per iteration sample complexity δ (Sa) remains the same as Theorem 4.1.
5 PRACTICAL IMPLEMENTATION OF ROLLIN
In this section, we describe the practical implementation of ROLLIN using Soft-Actor-Critte (Haarnjja et al., 2018) algorithm when an oracle curriculum is available. SAC can be seen as a variant of regularzee stochastic PG with the addition of the critics reduce the gradient variance. Recall that in the theoretical analysis, we learn a separate policy for each context that can start from the near-optimal state distribution of the previous context to achieve good return under the current

---

Under review as a conference paper at ICLR 2023
context. However, in practice, we usually would want have a policy that can directly start from the initial distribution to obtain a good return for the final context wk. In order to learn such a policy, propose to have two Tmain is the main agent that eventually will learn to achieve good return from p. and the second agent Teep is exploration agent that learns achieve good return under the current context from the near-optimal state density of the previous context. Another purpose of the exploration agent (as the name suggests) is to provide a better exploration experience for the main agent to learn the current context better. This is made convenient by using an off- off-policy RL agent where the main agent can easily learn from data that is not on-policy.
RL agents training in parallel, where the first agent
Specifically, for each episode, there is probability of 3 where we run the main agent conditioned on the previous context for a random number of steps until we switch to the exploration agent to collect experience for the current context until the episode ends. Otherwise, we directly run the main agent for the entire episode. Both agents are trained to maximize the return under the current context. Whenever the average return of the last 10 episodes exceed performance threshold Rt, immediately switch to the next context and re-initialize the exploration agent and its replay buffer. A high-level description is available in Algorithm 2 (a more detailed version in Algorithm 8).
Algorithm 2 Practical Implementation ROLLIN
1: Input: {ax | imput curriculum, R: near-optimal threshold, 8: roll-in ratio, H: horizon, 7
discount factor.
2: Initialize D t 0, Dexp + 0, k 0, and two SAC agents maain and Texp
3: for each episode do
4: if average return of the last 10 episodes under context w/ is greater than R then
5: A: + k+ Deep + 0, and re-initialize the exploration agent Texp
6: end if
7: if i >> 0 and with probability of /3 then Geomm1 - 7) (truncated at H) run Tmain (a|s Wk-1 1) from the initial state for h steps and switch to (axp wx) until the episode ends to obtain trajectory To:ı1 (5o, 4o, To: 81, 91,41; ... "., 881)
8:
9:
10:
record To.H in DD and Th:H in Dexp.
11: else
12:
Tmain (als, wx) to obtain trajectory To.H and record To.H in D.
13: end
14: at each environment step during the episode, update Tmain using DD and Texp using Dexp-
15:
end for
Output: Tiain
6 EXPERIMENTAL RESULTS
While the focus of our work is on developing provably efficient ap- proach to curriculum learning, we also conduct an experimental evalu- ation of our practical implementation of ROLLIN with soft actor-critic (SAC) (Haarnoja et al., 2018) the RI. algorithm on several continuous control tasks including goal reaching with an oracle curriculum and learned curriculum, well non-goal tasks.
x=1/3
6.1 GOAL REACHING WITH AN ORACLE CURRICULUM
We adopt the antmaze umaze environment (Fu et al., 2020) for eval- uating the performance of ROLLIN in goal-reaching tasks. In the oracle curriculum case, we use hand-crafted path of contexts, where each con- text specifies the location that the ant needs to reach (as shown in Figure
Figure Oracle curricu- lum of desired goals
2).' We consider a path of contexts w(x) parameterized by k E [0, 1] where (0) wi and (1) wk. and propose contexts along the path with a fixed step size A. See Appendix E. for more implementation details.
antmaze mamaaz--mazze
We combine ROL.IN with variety of prior methods, and we evaluate the following conditions:
(1) standard goal reaching; (2) goal reaching with goal relabeling (Andrychowicz al., 2017);
Note that in general, ROLLIN is not specific to goal-conditioned RI. and could in principle utilize any type of context-parameteriiee reward function that satisfies its assumptions.
7

---

Under review as a conference paper at ICLR 2023
(3) goal reaching with Go- Go-Explore (Ecoffet et al., 2019). For goal relabeling, we adopt similar relabeling technique as Pitis et al. (2020), where each mini batch contains 1/3 original transitions, 1/3 transitions with future state relabeling, and 1/3 transitions with next state relabeling. We implemented the Go-Explore method by adding an additional standard Gaussian exploration noise (multiplied by constant factor) to the agent for learning the next goal w + 1), once it reaches the current goal wk). We empirically observed that sampling the replay buffer from a geometric distribution with p 10-5 (more recent transitions are sampled more frequently) improves the overall performance. Hence, in all future experiments, we compare the performance of ROLLIN with classic uniform sampling and the new geometric sampling. We demonstrate how adjusting the ROLLIN parameter
(3 = 0.1, 0.2, 0.5, 0. 05, 0.9) impacts the learning speed on three different step sizes A We expect coarser curricula with larger steps to be more difficult.
I I
21 18 T2'
Main comparisons. We first provide overview experiments that compares ROLLIN with fixed
= 0.1 on different step sizes in different settings. In each case, we compare the prior method (vanilla, relabeled, or Go- (oo-Eplorr) with and without the addition of ROLLIN. As shown in Table 1, ROLLIN improves the largest value of x reached by the agent in most presented settings (except Go-Explore with A 1/12). This result suggests that ROLLIN facilitates goal-conditioned RL with curriculum, as we only update the learning progress x. to K + A when the return of the current policy reaches a certain threshold R (See detailed update of in Algorithm 2). Note that 3 0.1 does not always produce the best result, we will provide more results comparing different 3s in different settings later in this section, and we leave all the learning curves and detailed tables to Appendix F.I. Note that we do not include the results of directly learning the last context in the antmaze - e environment because the agent cannot reach the goal without the aid of a curriculum, which is corroborated by Pitis et al. (2020).
wlo Geometric Sampling
w/ Geometric Sampling
Setting
Method
1/24
1/12
A 1/24
A 1/12
Vanilla
Baseline
0.40 +: 0.02
0.36 + 0.00
0.82 + 0.08
0.38 1 0.03
ROLLIN
0.49 4: 0.04
0.44 1: 0.01
0.92
0.02
0.55
Relabeling
Baseline
0.89 +: 0.03
0.66 + 0.04
0.76 +: 0.02
0.72 1: 0.03
ROLLIN
0.91 1: 0.03
0.74 1: 0.01
0.78
0:01
0.73 0.00
Go Expllxe Noise = 0.1
Baseline
0.37 +: 0.02
0.38 4: 0.01
0.07
0.42 0.03
ROLLIN
0.52 = 0.07
0.38 + 0.01
0.93 + 0.02
0.43 0.02
Table Learning progress x at 3 million environment steps with varying curriculum step size A of different settings of goal reaching in antmaze -uaaze. We pick / = 0.1 for all experiments using ROLLIN. the results of using other Ss, As, and exploration noise be found in Table 5, 6, in Appendix F.1. The standard error computed over 8 random seeds.
6.2 GOAL REACHING WITH A LEARNED CURRICULUM In this section, we focus on the setting where an or- acle curriculum is not provided. In particular, we show that ROLLIN can improve MEGA (Pitis et al., 2020), an existing automated goal curriculum gener- ation method (Figure 3). MEGA proposes intrinsic goals that are achievable, but appear rarely in the replay buffer. Since there is no clear notion of pro- gression in the Icarned curriculum setting, we the progression in the number of environment steps where W/ corresponds to the goals during k 100K (k + 1) x 100K steps. We follow the same high- level procedure as described in Algorithm 2 where we randomly roll in policy that was trained on the previous context (in this case, would be the policy checkpoint that was obtained after the last 100K-step learning segment). A detailed algorithm procedure provided in Algorithm 9.
MEGA w/o Rollia
MEGA Rollin
Sncrms Race
0.75
0.25
0.00
Env. Stepe (x10')
Figure The success rate of goal reaching in antmaaze-umaz with a curriculum generated by MEGA with 9 0.5. The success rate of learned policy reaching the target goal is re- ported over 8 independent random seeds. 10 ran- dom trials for each seed are used to estimate the success rate. See Appendix E.2 for more details and results for other f8s.
6.3 NON-GOAL REACHING TASKS
For the non-goal tasks, we choose a fixed contextual space with ten discrete contexts: R. E (0.1, ..2,..,, 1), where each en uniquely determines the desired z-velocity of locomotion agent in
8

---

Under review as a conference paper at ICLR 2023
the following environments: walker2d, hopper, humanoid, and ant in OpenAI gym (Brock- man et al., 2016). Our goal is to train an agent to move fast and stably. For each context (H), we set the desired speed range to be [Ax, + 0.1)), where À is a parameter depending on the physics of the agent in different environments. When the z-velocity of an agent is within the desired velocity range, we set the healthy_rew rd higher value hea 1 sechy_eewrrdr w__hihhhhhhhhh Otherwise, we set the
reward to lower value healthy_reeeee Baarl__ewarr__ddee 1oo. In each environment, we increase the task difficulty with later curriculum steps (larger k), by increasing the near-optimal threshold R(x). Detailed parameters of the desired speed range X, near optimal-threshole R(x), healthy_reward. high, and
lw are provided in Appendix E.3.
Main comparisons. We first compare ROLLIN with fixed 3 0.1 at different environment steps: 0.75 x 100, x 10 In each case, we compare the learning progress K. averaged z- z-velocity, and averaged return, with and without the addition of ROLLIN. Note that for the case without ROLLIN, we still provide the curriculum to the agent for training. As shown in Table 2, ROLLIN improves the largest learning progresses K, average z-velocity, and average return in most presented settings. This result suggests that ROLLIN also can facilitate learning of non-goal tasks, as we only update the learning progress R. to Æ + 0.1 when the return of the current policy reaches a certain threshold R(x) (See detailed update of in Algorithm 2). Note that 3 = 0.1 does not always produce the best result, we will provide more results comparing different 3s in different settings later in this section, and we leave all the learning curves and detailed tables to Appendix F.3.
Step 0.75 100
Step 10
Mothod
2-velocity
renaum
a-velocity
rctura
wall kee
Scratch
n/a
2.90 +: 0.59
3937.6 6 + 120.6
n/a
3.26 : 0.36
4212.3 : 151.4
Baseline ROLLIN
0.89 4: 0.03
3.20 0.40
3838,2 1: 234.0 3831.7 133.3
0.92 0.03
3.69 :: 0.27
4032.3 à: 224.3
0.02
3.48 0.29
0.94
1 0.26
159.6
hoppez
Scratch
W/a
2.27 + 0.30
3081.0 93.5
2.02 + 0.12
3073.2
137.7
Baseline
0.86 = 0.02
2.48 0.14
3325.8 + 100.6
0.88 + 0.01
2.58 + 0.16
3386.2 + 124.7
ROLLIN
0.86 = 0.02
2.54 + 0.17
3339.7 + 151.4
0.89 4: 0.00
2.65 =i 0.15
$421.9 = 109.8
humanoid
Scratch
n/s
0.30 0.05
2737.0 = 85.5
nia
0.37 4 0.05
2763.8 96.5
Baseline
0.54 #: 0.01
0.32 1: 0.05
3007.5 +: 176.6
0.67 4: 0.03
0.39 d: 0.05
3017.2 2 4: 169.0
ROLLIN
0.67 + 0.03
0.39 0.08
3151.0 +: 171.6
0.69 1: 0.08
0.46 4: 0.09
3173.6 d: 238.3
ant
Scratch
n/a
4.36 + 0.38
3946.2 + 169.4
ala 0.00
4.45 # 0.38
4277.9 # 120.0
Baseline
= 0.02
4.30 0.37
3151.0
224.4
1.00
4248.5 88.6
RULLIN
0.99 = 0.01
4.50 d: 0.30
4242.1 = 89.5
1.00
0.00
4.66 =: 0.30
4473.0 = 102.2
Table 2 Learning progress K, average z-velocity, and average return at the 0.75 and 1.0 million environment steps in wall ker, hopper, hunanoi d, and ant. The average x-velocity and return are estimated using the last 50k time steps. "Scratch" shows the results of directly training the agent with the last context (1)). "Baseline" indicates 3 = O where we provide the curriculum ((t) to the agent without using ROLLIN. We pick 3 = 0.1 for all experiments using ROLLIN. the results of using other 3s can be found in Table 8, 9. 10 in Appendix F.3. The standard error is computed over 8 random seeds.
6.4 EXPERIMENTAL SUMMARY
In summary, we empirically demonstrated that ROLLIN improves the performance of goal-reaching tasks and non-goal tasks in different settings with vast range of selection of Bs. Although ROLLIN introduces an extra parameter 3 for finetuning goal-reaching tasks with a curriculum, our extensive experiments suggest that one could expect improvement by choosing a constant 3 0.1 or 0.2. Still, for achieving the best performance in general-tase one shall still consider fine-tuning B.
7 DISCUSSION AND FUTURE WORK
We presented ROLLIN, a simple algorithm that accelerates curriculum learning under the contextual MDP setup by rolling in a near-optimal policy to bootstrap the learning of new nearby contexts with provable learning efficiency benefits. Theoretically, we show that ROLLIN attains polynomial sample complexity by utilizing adjacent contexts to initialize each policy. Since the key theoretical insight of ROLLIN suggests that one can reduce the density mismatch ratio by constructing a new initial distribution, it would be interesting see how ROLLIN can affect other variants of convergence analysis of PG (e.g., NPG (Kakade, 2001; Cen et al., 2021) or PG in a feature space (Agarwal et al., 2021; 2020)). On the empirical side, our extensive experiments demonstrate that ROLLIN improves the empirical performance of various tasks beyond our theoretical assumptions, which reveals the potential of ROLLIN in other practical RL tasks with curriculum. Our experiments also suggest that one could expect improvement by choosing constant / = 0.1 or 0.2. Hence, another potential direction on the empirical side would be providing an automatic tuning method to select the best B3

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yasin Abbasi- Yadkori and Gergely Neu. Online learning in mdps with side information. arXiv preprint urXiv::4400 1401166662..,. 6812, 2014.
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 2019.
Alekh Agarwal, Mikael Henaff, Sham Kakade, and Wen Sun. Pe-pg: Policy cover directed exploration for provable policy gradient learning. Advances in Neural Information Processing Systems, 33: 13399-13412, 2020.
Alekh Agarwal, Sham M Kakade, Jason D Lec, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXiv: 113, 2019.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning, pages 166-175. PMLR, 2017.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. arXiv preprint arXiv: 2017.
OpenAl: Marein Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Karl Åström and Björn Wittenmark. Adaptive control. Courier Corporation, 2013.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for rein- forcement learning. In International Conference on Machine Learning, pages 263-272. PMLR. 2017.
Andrea Bassich, Francesco Foglino, Matte Leonetti, and Daniel Kudenko. Curriculum learning with progression function. arXiv preprint arXiv:2008, 00511, 2020.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pages 1471-1479, 2016.
Stav Belogolovsky, Philip Korsunsky, Shic Mannor, Chen Tessler, and Tom Zahavy, Inverse rein- forcement learning in contextual mdps. Machine Learning, 11009):225--3344 2021.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 2019.
Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. arXiv preprint arXiv: XXXv:/906600/1 0076007766, 2019.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint vrXiv:1100.2894. 2018
Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research, 2021.
10

---

Under review as a conference paper at ICLR 2023
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta-policy optimization. In Conference on Robot Learning, pages 617-629. PMLR, 2018.
Christoph Dann, Lihong Li, Wei Wei, and Emma Brunskill. Policy certificates: Towards accountable reinforcement learning. In International Conference on Machine Learning, pages 1507-1516. PMLR, 2019.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in Neural Information Processing Systems, 33:13049-1306) 2020.
Carlo D'Ermo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowledge in multi-tase task deep reinforcement learning. In International Conference on Learning Representa- tions, 2020. URL https: 1ttp://rwwweeeetttoooooooeee/eee itreruunndddrggg222222
net foree//ooummidie
Yuhao Ding, Junzi Zhang, and Javad Lavaei. Beyond exact gradients: Convergence of stochastic soft-max policy gradient methods with entropy regularization. arXiv preprint arXiv:21 2021.
Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning, pages 1665-1674. PMLR, 2019.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for hard-exploration problems. arXiv preprint arXiv:1901. 190111999999 2019.
Ben Eysenbach, Xinyang Geng, Sergey Levine, and Russ R Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement. Advances neural information processing systems, 11114783-14999 2020a.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning achieve goals via recursive classification. arXiv preprint WIXv:20011008909 2020b.
Kuan Fang, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Adaptive procedural task generation for hard-exploration problems. arXiv preprint urXiv:2007.00355 2020.
Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In International Conference on Machine Learning. pages 1467-1476. PMLR, 2018.
Chelsea Finn, Pieter Abbecl, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
Carlos Florensa. David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. Conference on robot learning, pages 482-495. PMLR, 2017.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. International conference on machine learning, pages 1515-1528. PMLR, 2018.
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. arXiv preprint arXiv: 1686, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Yasuhiro Fujita, Kota Uenishi, Avinash Ummadisingu, Prabhat Nagarajan, Shimpei Masuda, and Mario Ynocente Castro. Distributed reinforcement learning of targeted grasping with active vision for mobile manipulators. 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9712-9719. IEFE, 2020.
11

---

Under review as a conference paper at ICLR 2023
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference Learning Representations, 2021. URL https: http://pnnrvieeeeeeeeeeeeeeeeeee id=rALA0Xo6yNN.
Graham C Goodwin and Kwai Sang Sin. Adaptive filtering prediction and control. Courier Corpora- tion, 2014.
Anirudh Goyal, Riashat Islam, DJ Strouse, Zafarali Ahmed, Hugo Larochelle, Matthew Botvinick, Sergey Levine, and Yoshua Bengio. Transfer and exploration via the information bottleneck. In International Conference on Learning Representations, 2019, URL https::t////ttt //op:r/ee//eerreiie.. net/ fto
Anirudh Goyal, Yoshua Bengio, Matthew Botvinick, and Sergey Levine. The variational bandwidth bottleneck: Stochastic evaluation on an information budget. International Conference on Learn- ing Representations, 2020. URL https: /tttt::://opprreeee /wtp///vvie///eo////////p net www.:eenttooooeeeee forum Pid=Hyelllee kIVFDS.
Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham M Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. arXiv preprint arXiv:2210.09579, 2022.
Tuomas Haarnoja. Acquiring diverse robot skills via maximum entropy deep reinforcement learning. University of California, Berkeley, 2018
Tuomas Haaroja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/2.0599 2018.
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv: TrXx:::502.02259 2015.
Kristian Hartikainen, Xinyang Geng. Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for sumi-supervieed and unsupervised skill discovery. arXiv preprint arxiv: 1907,0822. 2019.
Boris Ivanovic, James Harrison, Apoorva Sharma, Mo Chen, and Marco Pavone. Barc: Backward reachability curriculum for robotic reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 5-- IEEE, 2019.
Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. Contex- tual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning, pages 1704-1713. PMLR, 2017.
Chi Jin, Zeyuan Allen Alu, Sebastien Bubeck, and Michael I Jordan. q-learning provably efficient? Advances in neural information processing systems, 31, 2018.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning. Citeseer, 2002.
Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski.
Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic rein- forcement learning at scale. arXiv preprint 1rXiv:2104.08212, 2021.
Tae-Hoon Kim and Jonghyun Choi. Screenernet: Learning self-paced curriculum for deep neural networks. arXiv preprint
1001:00000000 2018.
Pascal Klink, Carlo D' D'Eramo, Jan R Peters, and Joni Pajarinen. Self-paced deep reinforcement learning. Advances in Neural Information Processing Systems, 33:9216-9227, 2020.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
12

---

Under review as a conference paper at ICLR 2023
Iyy Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 10 2021.
URL htttps://githeie /githubtth.h.eegg com/ http///oosttttoaa////o jaxri.
Tejas Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv:1606.0255: 2016.
loan Doré Landau, Rogelio Lozano, Mohammed M' Saad, and Alireza Karimi, Adaptive control: algorithms, analysis and applications. Springer Science & Business Media, 2011.
Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Softmax policy gradient methods can take exponential time converge. In Conference on Learning Theory, pages 3107-3110, 10. PMLR. 2021.
Xingyu Liu, Deepak Pathak, and Kris M Kitani. Revolver: Continuous evolutionary models for robot-to-robot policy transfer. arXiv preprint arXiv:2202.052444 2022.
Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference on Machine Learning, pages PMLR, 2020.
Russell Mendonca, Oleh Rybkin, Kostas Danilidiis, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals via world models. Advances in Neural Information Processing Systems, 34, 2021.
Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. Kinematic state abstrac- tion and provably efficient tich-observation reinforcement learning. In International conference on machine learning. pages 6961-6971. PMLR, 2020.
Aditya Modi, Nan Jiang, Satinder Singh, and Ambuj Tewari. Markov decision processes with continuous side information. In Algorithmic Learning Theory, pages 597-618. PMLR, 2018.
Aditya Modi, Nan Jiang, Ambuj Tewari, and Satinder Singh. Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics, pages 2010-2020 PMLR, 2020.
Ted Moskovitz, Michael Arbel, Jack Parker- Holder, and Aldo Pacchiano. Towards an understanding of default policies in multitask policy optimization. In International Conference on Artificial Intelligence and Statistics, pages 10661-10686. PMLR, 2022.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances neural information processing systems, 30, 2017.
Ashvin Nair, Shikhar Bahl, Alexander Khazatsky, Vitchyr Pong. Glen Berseth, and Sergey Levine. Contextual imagined goals for self-supervised robotic learning. In Conference Robot Learning, pages 530-539. PMLR, 2020.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018
Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303-1315. PMLR, 2022.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. arXiv preprint arXiv:2003.04960, 2020.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. In ICMI,, volume 99, pages 278-287, 1999.
13

---

Under review as a conference paper at ICLR 2023
Shayegan Omidshafiei, Dong-Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christopher Amato, Murray Campbell, and Jonathan How, Learning to teach in cooperative multiagent reinforcement learning. In Proceedings of the AAAI conference artificial intelligence, volume 33, pages 6128-6136, 2019.
OpenAl. Openai five. betps://bles htt.:/tcpp...oogo.oppaaaacccooaa wttppeaacco/opeeaaaiiiiiiiii...a
2018.
OpenAl OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, llge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D'Sa, Arthur Petron, Henrique P O Pinto, al. Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv preprint arXiv:2101.04882 2021.
Jack Holder, Minqi Jiang. Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. arXiv preprint 2rXiv:2203.01000 2022.
Jan Peters and Stefan Schaal. Natural actor-critic. Neurocomputiing, 1117-991110--1900 2008.
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference Machine Learning, pages 7750-7760 PMLR, 2020.
Vitchyr H Pong. Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering sele-supe reinforcement learning. arXiv preprint arXiv: 0ete:/933009999 2019.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning, pages 835-853. PMLR, 2020.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint 161.v.1610.01283, 2016.
Zhizhou Ren, Kefan Dong. Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. Advances in Neural Information Processing Systems, 32, 2019.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukeuoglu, and Raia Hadsell. Policy distillation. arXiv preprint arXiv: 2015.
Shankar Sastry, Mare Bodson, and James F Bartram. Adaptive control: stability, convergence, and robustness, 1990.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference machine learning, pages 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1777.063:0 2017.
Shamane Siriwardhana, Rivindu Weerasakera, Denys JC Matthies, and Suranga Nanayakkara. Vusfa: Variational universal successor features approximator to improve transfer drl for target driven visual navigation. arXiv preprint arXiv: /908. 06376, 2019.
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint 1eXivv1507.0033 2015.
Sainbayar Sukhbaatar, Emily Denton, Arthur Szlam, and Rob Fergus. Learning goal embeddings via self-play for hierarchical reinforcement learning. arXiv preprint arXiv: 2018.
Wen Sun. Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal. and John Langford. Model-based in contextual decision processes: Pac bounds and exponential improvements over model-free approaches. In Conference learning theory, pages 1998-2933..2 PMLR, 2019.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
14

---

Under review as a conference paper at ICLR 2023
Fumihide Tanaka and Masayuki Yamamura. Multitask reinforcement learning on the distribution of mdps. In Proceedings 2003 IEEE International Symposium on Computational Intelligence in Robotics and Automation. Computational Intelligence in Robotics and Automation for the New Millennium (Cat. No. 03EX694), volume 3, pages 1108-11 13. IEFE, 2003.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pages
2017.
Gang Tao. Adaptive control design and analysis, volume 37. John Wiley & Sons, 2003.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):550-354 2019.
David Warde-Farley Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint arXiv:1811.11355 2018.
Ronald J Williams.
Simple statistical grodient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.
Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, Yinyuting Yin, Bei Shi, Liang Wang, Tengfei Shi, Qiang Fu, Wei Yang, Lanxiao Huang, and Wei Liu. Towards playing full moba games with deep reinforcement learning. In H. Larochelle, M. Ranzato, Hadsell, M. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 621-632. Curran Asso- ciates, Inc., 2020. URL 06d5ae1 0 bbe
wewpeeeddnnnnnuuuingneeeeeeeeeee neurips wet.iis.cccppppee cccpppppr//22//FF.cccccccc pdf.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836, 2020a.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pages 1094-1100 100. PMLR, 2020b.
Rui Yuan, Robert M Gower, and Alessandro Lazaric. A general sample complexity analysis of vanilla policy gradient. In International Conference on Artificial Intelligence and Statistics, pages 3332-3380. PMLR, 2022.
Yuexiang Zhai, Christina Back, Zhengyuan Zhou, Jiantao Jiao, and Yi Ma. Computational benefits of intermediate rewards for goal-reaching policy learning. Journal of Artificial Intelligence Research, 73:847-896, 2022.
Junzi Zhang, Jongho Kim, Brendan O' O'Donoghue, and Stephen Boyd. Sample efficient reinforcement learning with reinforce. arXiv preprint arXiv:2010.11364 page 97, 2020a.
Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence policy gradient methods to (almost) locally optimal policies. SIAM Journal Control and Optimization, 58(6): 3586-3612, 2020b.
Tianjun Zhang, Benjamin Eysenbach, Ruslan Salakhutdinov, Sergey Levine, and Joseph E Gon- zalez. C-planning: An automatic curriculum for learning goal-reaching tasks. arXiv preprint 10 2021.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. Advances in Neural Information Processing Systems, 33:7648-7650 2020c.
Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learningvia reference-advantee decomposition. Advances in Neural Information Processing Systems, 33: 15207, 2020d.
15

---

Under review as a conference paper at ICLR 2023
APPENDIX A GENERALIZATION BETWEEN DIFFERENT TASKS IN THE CONTEXT SPACE
A.1 SUMMARIES OF NOTATIONS AND ASSUMPTIONS
1. The maximum entropy RI (MaxEnt RI.) objective with initial state distribution p rein- forcement aims at maximizing (Equation 15 & 16 of Mei et al. (2020))
(T(p) =
Eso (aulsn) [r(s), a;)] aHH(pp T))
(4)
and (a [s,)) is the discounted entropy term
H(p, T)
E
--n log (anlss))
(5)
60 C0T~p.0& wes ( (-18a)
L==
and a is the penalty term. For simplicity, we denote the optimization objective function in (4) a-MaxEnt RL. Similar to Equation 18 & 19 of Mei et al. (2020), we also define the advantage and Q-functions and for MaxEnt RL as
(s (s, a) log ((s a) (T)),,
(=,s, a(s + Σ P(s'ls,
(6)
2. We let
d% (8) = (1- ? P = s|80),
(7)
1=0
to denote the discounted state visitation of policy π starting at state so, and let
(,)()) = Ex-pplp [df (s)]
(8)
denote the initial state visitation distribution under initial state distribution p.
3. We assume the reward functions under all context are bounded within [0, 1]:
(w a) E [0, 1], Vi 22 V(s a) S A.
4. Similar previous analysis in Agarwal al. (2021); Mei al. (2020); Ding al. (2021), we assume the initial distribution for PG/stochastic PG satisfies p(s) > 0, Vs e S.
A.2 MAIN RESULTS: MISMATCH CORFFICIENT UPPER BOUND
Lemma A.1 (Density Mismatch Ratio via ROllIN) Using (1) from ROLLIN μs
Bd
Tis du
(1 - p) the density mismatch ratio
/Pk
satisfies
dps
S
a(l - 3) A:s)
L
(10)
Pk
where A5 = maxisigs |ww - Wi- 1112-
Proof By (1) from ROLLIN, we have
Tizz- Fwa-1
dyt
dpti"
dpa= d
1/
Jum
dp
(1I)
min p4
3d 8d + (1 - B)p
Tx =
dpw
min M4
B
16

---

Under review as a conference paper at ICLR 2023
where inequality (i) holds because of (1), and inequality (ti) holds by setting p 0. Now it remains
to bound d - dxi
using the difference || - wkllz ll2- Let P = = (s/[sp pu) denote
1 the state visitation distribution resulting from Tzy probability starting at jig. then we have
(5') P (5'
Σ
[ (s) (a|s) ( P(s'|s, a)
2.0
ph h (8) (als) ph (s) (als) (s) (als) - (5) (ala)) P(s'(s,))
8,0
pk. (s))
Σ Tun (als) - (als)) P(s"|s.a)
Σ (alss(s)
+ Σ (1P 11)) (+))
(12)
Taking absolute value on both side, yields
| - pp--111 1 = Σ P (s') - 1 (s')l
P - 1 (s) Σ
Tume (als) - Tuدn (a|s)
Σ ((s)
Sen liwa wise 1ll2
(13)
+ Σ 1 (s) - -s) ΣΣ Tue (als))
(
Sc 1lwk - wk-11l2 + I P 1lll ... S sch llws - wk wk-1ll2 + ||lt P/- '|1
(1) c ||wk - wk-1ll2 + ||
1
where inequality (i) holds by applying Lemma B.2 with e₁ = L,/a(1 - 7) from and equality (ii) holds because the initial distribution of 1Pt is uaa By the definition of di, we have
dfv (5) d (s) (x()) da 1 = (1 (s)
7) 7 (P)
( Vs E. (14)
where in equality (i), we use du to denote di. Adding ll norm on both sides of (14) and applying (13), yields
||dg- de-1ll1 11-1/115111 (1--) 7) Σ x (eahllus wk-1ll2 + I1 1x-1111)
h=0
(15)
( 701 We wk-1ll2 + lle 1012
(ii) 701
|lwk wk-1 112 + 3 [|ds- de-21l1
1 1-7
where equality (i) holds because En Hence, we know that
= >//=1(1 7)2 and equality (ii) holds because of (1).
|d dr 1
7C1
| Wk-1 122 1l dx-2l1
1 7
711
Σ
||xiا1 willz gh- - ||d doll
(16)
1
1=0
701
A + gh ||d1 doll ~ (1- 71 33)
701
1- 3
17

---

Under review as a conference paper at ICLR 2023
where A six ||w;
w1-1ll2 and the last holds when i is large. Therefore, applying
(16) back to (11), we know that
Fun dpc
dp
dyss*
Pk
min M4
B
(17)
(6)
1
7C1
+
52
aa11 - B) A
Lr
min Pak (1 7) 3)
3
where inequality (i) holds since Lemma B.2 implies c = Trr/a(11 - 7), and we omit the 1/(1 - 76 and log in the Ω. which completes the proof. Note that we can only achieve the final bound ( A by setting 3 as a constant. If we pick an arbitrarily small B3, then the 1/3 term will Ls dominate the complexity and we will not have the final bound of (ottoy) A.
A.3 COMPLEXITY OF VANILLA STOCHASTIC PG
Theorem A.2 (Complexity of Stochastic PG (Theorem 5.1 of Ding et al. (2021))) Consider an arbitrary tolerance level > O and a small enough tolerance level & 0. For every initial point 01, if t1 is generated by SPG (Algorithm 4) with
SL
T >
6D()))
C 1n
T3
Eu
1) to: == T2,
8Eo
6de
300
6oT log Ti
0² #ln[T2 + to)
B1 max
B2
(18)
Cgend'
ÂL.
60ede
loog Ti
8
1
Th = 17 S min
V1 < t $ Ti, TTt
Vt > Tí,
TiL
C 2L
Ti +
where
D(0.)= = yx (p) - -V*0 (p), 80 = min {
(aminess pls) 2 6ln2
1 7) 7)
(-
<exp
.1
302
2a
dm
10
C
min p()) min min (o(a)s)?
2000
p
SES DE00 s.a
2x
Cc
min p(s)(1 ²) min x (alss)
S
P
SEg
00
0 E
min c
11-00'1 S (1 + +1/1)/
A |loo co,, log π "1'1"
8
11 (a (1+(alogg log A)2
8 a(4 8 8log A)
Co inf min To, (als), 0²
( - 1/2)2
L
(21
(1 7}²
(1 71-7)3
(19)
then we have P(D(fr+1 1- 8.
Corollary A.3 (Iteration Complexity and Sample Complexity for e-Optimal Policies) Suppose we set the tolerance level E, Ó(S-1), the iteration complexity and sample complexity of obtaining an e-optimal policy using stochastic softmax policy gradient (Algorithm 4) in phase I and phase 2 satisfies:
* Phase 1: Ti = 01 (522*)
B1 = (525*)
Phhase 2: T2 ($3/22)
B₂ . (S*).
18

---

Under review as a conference paper at ICLR 2023
with probability at least 1 5.
Proof We first check the dependency of (19) on S. Notice that
Egi
)² <eepp (mminesppss)
20())) { {aminnes p(s)
In
max
(7
7)
.11
.1 =
52(8));
(20)
lo:
302
to 2
52(S):
(21)
20ED
. CC.
1
8
d=
max p(s)
mingegs min To(a 5((53):
(22)
C8
2a
P
8ES
Ce:
1
S
d
max p(8) (1 -)) max x (alls=²=((($3) EES
(23)
C
2a
Hence, the complexities in phase scales at
Ti 2
(b)()))
= ($250)
Bi max
30g² Cy
GoT, log Ti
-
E (S257).
(24)
den
AL
To enforce positive T2, the tolerance level e, 5 should satisfy
1, which implies मे N(S2).
68g
Hence, if assuming _ = o(S)) the tolerance level e, d = O ) the complexities in phase 2 scales at
E0 -1 -1)ta lo = 50 (5312). B₂ 2 o² In(T2 to) 50 (5*).
T2 >
(25)
60yde
A.4 COMPLEXITY OF LEARNING THE NEXT CONTEXT
Theorem A.4 (Theorem 4.1: Complexity of Learning the Next Context) Consider the context- based stochastic softmax policy gradient (line 7 of Algorithm 1), suppose Assumption 3.1 and Assumption 3.2 hold, then the tteration number of obtaining an e-optimal policy for wh. from 0" 0-g-1
Ō (83/2) and the per iteration sample complexity is 52 (
We first introduce the following lemma to aid the proof of Theorem A.4.
Lemma A.5 (Bounded Optimal Values Between two Adjacent Contexts) Under the same condi- tions as Theorem A.4, we have
Tiz-1 Vun
2L, | wk-1 lg
(0)
(p) <
(26)
(1 7
Proof Let V denote the value function of policy & with reward function Tw. From (65) of Lemma B.3, we know that for any initial distribution p. we have
Thz-1 (p) =- (p) Vi
Σ
-rz- de
(s) a DKI. (nir. (+ (15))
(27)
19

---

Under review as a conference paper at ICLR 2023
From (47) of Lemma B.1, we know that
exp Q 1 (8.0)/a
Twp_, (als) co
Ev exp QTh (s,
(28)
exp (s.
T2a (als) (ooftmax(Q"=
E. exp [M (s, as,aa//aa
hence, we have
DKL (iin- 1 (= (-|s)
= Σ Tux I (a|s) (og ([softmax ( - (anttmax("mn-1,000)0))))))
(a,s)
log ([softmax(Q (a. (a. s)/aa)])
(29)
Let f(x) denote the log soft max function for an input vector æ [z], ........ x xAl _ such that
I; 0, then for small perturbation A RA,, the intermediate value theorem impliee
[lf( + A) (==),, ه V. [f())),
(30)
for some vector on the segment [x, x /
A Now consider the Jacobian of the log softmax function
0[V=f(z)
1 - (=) E 1) E
if j. otherwise,
(31)
0zg
( (-1,0)
where p,(2) exp(zi)/ / (xp( hence, we know that
@[f())]
[[f( + A) )[. - [f(x)), A v=
SIA
8zk
(32)
= -|| 20
1-pi(z)+ + Eps(z) 21
Now let
2 - 1Qth 1 (1, Q 1 as, Q (s,
(33)
A- th (s, a1), = (s, Q (s, [{))
(57) from Lemma B.2 implies that
1 Q 1
L, |aw - wk-11l2
(34)
a(l -7)
substituting (34) and (32) into (29), yields
< 2t 2TT (dls) 21 5 2L, l|wk - Wk Wk-1llz
DKL (Tin_ (-1s)][=2, (-1s)
(35)
a(1 71)
Combine (35) with (27), we have
Vin (p) Vil - (p) = 1-7 Σ
dg wu (s)
1 a DKL. (nin. (115) (115))
(36)
2I | wk-1 l12
(1 - 7
which completes the proof.
20

---

Under review as a conference paper at ICLR 2023
Now are ready to proceed to the proof of Theorem A.4. Proof
From (19) we know that
(a [aminness { p(s)
2
<exp (-a1
1
1
1
6o min
1
(37)
6ln
52
And from Section 6.2 of Ding et al. (2021), we can directly enter phase 2 of the stochastic PG when
V (p) - Twn (p) S EO.
(38)
Hence, when A maxigisk ||w: W; w1-1112 O(1/52), we have
Vax (p) Vin Juk-1 (0) 5 2 Eu (1 7) 2
(39)
which implies we can directly enter phase 2 and enjoys the faster iteration complexity of T2 Ω (53/2) (by choosing O(S)))) and the smaller batch size
o² IT2 + to)
L
L 3)
B2
52
52
sa
(40)
60ede
a(1
a(1
where equation (i) holds by applying Lemma A.1 (23):
o² In(2 to) 52 5a
J uu 1
Lr
= 8
6Cede
a(1 - B)
and equality (it) holds by the assumption that A = 0(S-2) and we omit the log term and components not related to S in f2.
A.5 TOTAL COMPLEXITY OF ROLLIN
Theorem A.6 (Theorem 4.3: Total Complexity of Learning the Target Context) Suppose As- sumption 3.1 and 3.2 hold, and 0 is near-optimal initialization, then the total number of iteration of learning Tww using Algorithm I (3/22) and the per iteration Qt (8"), with high probability.
Proof From lemma A.5, we know that
2Lr ||w Wk-1 l2
V (p) Vinui (0)
(41)
(1 7
Suppose for each context wa. initialize the parameters of the policy as 0(h) 0z.-, and let a(*)) denote the parameters at the tt iteration of SPG. We will use induction to show that when 1 = (($3/2), vk E [K], we have
Fack-1) (p)
V (p) Vay
Eo: <
(42)
this implies that for any context WE, A E [K], we can always find good initialization by setting 8(4) -1) from learning TH, using SPG after 12(5322) iteration. This result guarantees
that every initialization 0(4) for learning the optimal contextual policy Th will directly start from efficient phase 2.
Induction: & == O. When A 0, Assumption 3.2 and the near-optimal initialization (Definition 4.2)
of (()) implies that
(p) V.n 7 (p) Eg-
(43)
This result implies that a near-optimal initialization allows the initialization to directly start from phase 2 of SPG.
21

---

Under review as a conference paper at ICLR 2023
Induction: from k 1 to k. Suppose the result in (42) holds for is 1, then we know that
Twp-1 (p)
(p) Vule -he-
(p) Vil 2) (p) Eo.
Vise
(44)
Select such that S Eu/2. Theorem A.4 suggests that when t' = (2(55/3), with high probability, we have
Vux
The (p) - Vax"
(p) <ES
ED
(45)
2
Hence, we initialize 0(4) = 0!h-11, with high probability when 1' 52(558/2) we have
Vi (p) - Vwn
(p) = Twx (p) Vwi (()) + Vux ((p) Viz
(p)
(46)
(8) + Vil (p) Vus (0(2-1) 1)
(36)
(0) 80,
where inequality (i) holds by equation (39) in Theorem A.4, inequality (ii) holds because of the induction assumption in (45).
Therefore, we have shown (42) holds for = 2(S3/3), Vk [K]. Since we have contexts in total, we know that Algorithm can enforce good initialization ofth) that directly starts from phase 2 for learning all Tun. and for each & [K]. the iteration complexity is (2(59/2). Hence the total iteration complexity of obtaining an s-optimal policy for the final context wK is 52 (KS5/2), with per iteration sample complexity of 52 (S*)
APPENDIX B KEY LEMMAS
B.1
OPTIMAL POLICY OF MAXIMUM ENTROPY RL NACHUM ET AL. (2017)
Lemma B.1 The optimal policy x² that maximizes the o-MaxEnt RI. objective (4) with penalty term a satisfies:
exp (pp(2=((2 (s,
x (a(ass exp (s.0) (
(47)
Eo exp (S8'((()s
for all l E N, where
(5, a) = r(8, + Pig V(5')
(48)
Vxa (8) = a log Σ exp (Q (s,))aaa)
Proof Similar proof appears in (Nachum et al., 2017), we provide the proof for completeness. At the optimal policy Th = n², take the gradient of (4) w.r.t p E A(A) and set it to 0, we have
Ep(a) p(a) (s, anp(aa) ( LaEA
Q² (s. Inp(aa 0,
(49)
Öp(a)
which implics
p(a) = exp ( (s. (0-(3,0) x exp (QT
(50)
Hence, we conclude that T (als) x (xp
B.2 BOUNDING THE DIFFERENCE BETWEEN OPTIMAL POLICIES
Lemma B.2 Suppose Assumption 3.1 holds, let T (a)s), T (als) denote the optimal policy for
a-MaxEnt RL (47), then V(s, a) E x A. the optimal policies of a-MaxEnt RL under context w, w' satisfy:
L ||W w/ll2
(als) - <
(51)
a(1
22

---

Under review as a conference paper at ICLR 2023
Proof From Lemma C.1, we know that the soft value iteration
TQ(s, r(s, a) yaBs'
log Σ Exp o'
(52)
is contraction. Let Qu: Ow denote the Q functions at the 1th value iteration under context w, w/ respectively, we know Q Q and Q Q Let = 11Q Qu blo then we have
Ei+1 = -1 - Q 1
log Σ exp (s'ss,
(s (rw((s,a) a) + yaEs'
yaB log
exp
< </ro - loo + ya Eg log E exp (s' aa)/a 1//- Ey log Σ exp Qu (s',
Slrr rw boo + + 12 = 1l - -lu + yEe.
(53)
where the last inequality holds because (ax) log Ei= exp(x;) ) is contraction. From (53), we have
........ Ex+1 & 1| - Iw +7E I (1+?) 1|r - -rr + ... S m|r - ro 1la0
E
? + E1
1
(54)
which implies
|I ro loc I
Top || w'll
@ - @ -
500 <
(55)
1 1-7
1-7
where the last inequality holds by Assumption 3.1. Hence, we have
(= Q (s,
Ly || - w'llz
Vs,
(56)
a(1
which implies
1 - Qu
Lr ||w -wll-- w'll2
(57)
a(1 77)
Next, let n,, Thr denote the maximum entropy policy RL under context w, w/ respectively, Then for fixed state action pair (s,a) E S x A, we have
exp (s
(als) (sfttax(((
Σn exp
(58)
exp (QF
(als) = (sftmax(Q"* (1
Σ. exp 1Q (s,
where (-,(s,)) Q (: E RA,, and we want bound | - - (a|s)]. Next we will use
(57) to bound |T] (a|s) Th (a|s)], where the last inequality holds by (56). Let f(x) denote the
softmax function for an input vector = I2,, ..... xA] such that 2 0, then for small
perturbation E A the intermediate value theorem implies
[[ + A),, [f(=)] = T V. [f())),
(59)
for vector the segment [ar, + A]. Hence
[[{())]
[ff( A),, - [f(x)), A [V>f(>)] < I|Aا|
Ozy
k=1
< || x pi(z)) + Σ (=))(((())
< 11
ps(z) + Erイ(2)
=||A||
J#1
(60)
23

---

Under review as a conference paper at ICLR 2023
where the Jacobian of the softmax function 0 d[V=f(z)), /0z, satisfies:
b [v=[(=))
[p:(2)(1-p))) ([a))1-pi(e) pi()))
=
(61)
Ōzj (p:(2)p;(z)
otherwise,
and p:(z) = exp(zi)/ Ex= k=1 exp(2k). Now let
1 (s, a1) (z) ( (8,aa)))
(62)
A
10 (s, ai) Q (8, 02), ..... @ (s, a사)l
We know that f(x) n (a|s) and f(x + A) = オ. (a|s). Then (57) implies that
L || w'l2
(63)
a(1 7)
substituting this bound on A into (60), we have
L, || wlll
(als) x [f(x) - f(x + A) IA S
(64)
a(1 7)
which completes the proof.
B.3 SOFT SUB- QUN-OPTIMALITY LEMMA (LEMMA 25 & 26 OF MEI ET AL. (2020))
Lemma B.3 For any policy a and any initial distribution p. the value function (p) of the a-
MaxEnt RL (48) satisfies:
v (p) (p) - - Σ (15(s) DEL (+(1))][(((s)s)
(65)
where x² is the optimal policy of the o-MaxFnt RI. (4).
Proof Similar proof appears in Lemma 25 & 26 of Mei et al. (2020), we provide the proof here for completeness.
Soft performance difference. We first show soft performance difference result for the MaxEnt value function (Lemma 25 of Mei et al. (2020)). By the definition of MaxEnt value function and Q-function (4), (6), VIT, n', we have
v (s) - (s)
Σ r'(ss) [Qr (s
- -aao)-alop alg ='ggs(( Σ T(ass)) (s, a log (a]ss)
(r'(ass) - x(als)) (0r log
+ <alss) [ (s, -s
a alg (als) (s, aalg))
Σ (r'(a))) - x(als)) (0F - ((als)))) + +aDKL
(66)
T(ass) Σ P(s(|s,a) [v) (
+
Σ (8') Σ (r'(aa's')- - a(a'(())) 10x (6',a') a log 7
+ +aDKI (DK (T(1|')||
24

---

Under review as a conference paper at ICLR 2023
where the last equality holds because by the definition of state visitation distribution
d% (8) (1
Σ ==
1P (st s|80),
(67)
taking expectation of 8 w.r.t. & ~ P. yields
V (p) (p)
1
Σ (''(''' - x(dls')) Σ d7(6')
( (s',a') a log x (a'ls')
(68)
+ aDRL (1(()
and (68) is known as the soft performance difference lemma (Lemma 25 in Mei et al. (2020)).
Soft sub-optimality. Next we will show the soft sub- sub-optimality result. By the definition of the optimal policy of o-MaxEnt RL (47), we have
a log " (als) Q" (s,a) V (s).
(69)
Substituting xF into the performance difference lemma (68), we have
(s) (s)
Σ (s')
Σ (F'('')s')
- -((ds)) [Q= (s', log x-' (d'ls')
=V"'(s''
+ aDL. (x(-|6')||= (+(s'))
(70)
Σ (s)))
(a'(a'))))) x((1(s()) v=' (5') + aDL (3((15))1==
(8)) ar DDL (-1))))(>('1'))))
taking expectation yields
V (p) (T)
Σ (s) a- DKL (T(11s)]lm ((|s))
(71)
which completes the proof.
APPENDIX C SUPPORTING LEMMAS
C.I BELLMAN CONSISTENCY EQUATION MAXENT RL
Lemma C.1 (Contraction of Soft Value Iteration) From (48) and (6), the soft value iteration op- erator defined as
(Q(,,a) = r(s,a) + yal.s
log Σ
exp (Q(s', a')/a)
(72)
a'
contraction.
25

---

Under review as a conference paper at ICLR 2023
Proof A similar proof appears in Haarooja (2018), we provide the proof for completeness. To see (72) a contraction, for each (s, a) E S A, we have
(1(5,a) T)) - r(s, rar Σ exp
a
Q2( + 11Q1 Q21lxc)
<r(s,0) + logog Σ cxp
(73)
|1Q (1Q11- 02l120
$r(s,a) + +aa exp
Σ exp
a'
(22(s,0)
=y -7/ Q21loo + +r(,, + ta log Σ exp
17/111 Q21lo + TQ2(s,a),
which implies TQi a) - TQ2(s,a) S ||Q Q21lxa Similarly, we also have TQ2(s, (Qi(s,a) [1Q Q21l hence conclude that
(8) Q2(8. a)) - ITQQ TQ2ll V(s,a) E A,
(74)
which implies ||Q] Q211a - 7 TQQ - TQ2 lax Hence T a 7-contraction and the optimal policy n² of it is unique.
C.2 CONSTANT MINIMUM POLICY PROBABILITY
Lemma C.2 (Lemma 16 of Mei et al. (2020)) Using the policy gradient method (Algorithm 3) with an initial distribution p such that p(s) 0, VS, we have
c: inf min To, (a|s) > 0
(75)
constant that does not dependent t.
Remark C.3 (State Space Dependency of constant c) Note that assuming is independent of is quite strong and does not generally hold as suggested by Li et al. (2021). Still, if one replace the constant with other S dependent function f(S), one still can apply similar proof technique for Theorem 1 to show that ROLLIN reduces the iteration complexity, and the final iteration complexity bound Theorem 4.1 will include an additional f(S).
APPENDIX D SUPPORTING ALGORITHMS
Algorithm 3 PG for a-MaxEnt RL (Algorithm in Mei et al. (2020))
1: Input: p, 0, 17 0.
2: for = 0, ",...,T T do
(p) 3: 0.+1 + 0, + 1 80,
4: end for
Algorithm 4 Two-Phase SPG M-MaEEnt RL (Algorithm 5.1 Ding et al. (2021))
Input: p. 0,0g, a, B1, B2, Tı, {m) for = 1. T do
2:
3:
if < Ti then
4:
B ==1 Bi
D Phase 1
5:
else
6:
B BB
Phase
7:
end
8: Run random horizon SPG with p, a. 0c, B, 1
Algorithm
9:
end for
26

---

Under review as a conference paper at ICLR 2023
Algorithm 5 Random-horizon SPG for c-MaxEnt RL Update (Algorithm 3.2 in Ding et al. (2021))
1: Input: 0, 0o. B, 6,
2: for 1,2, .... do
3: $'1, aH,, SamSA(p.8t.1)
Algorithm
4: @ f EstEntQ H d' 02, 7,
b Algorithm
5: end for
Di=1 6: 0et1 0s (1-7)8
dee B
Vo log no, 1611,1111 (Qra,* a log no, (aiz, 1sh).)
Remark D.1 Lemma 3.4 in Ding al. (2021) implies that the estimator
(1->) Vo log ne, (a'n ss,,) - a log Te. a
(a'n, 1s2.))))))
(76)
in line 6 of Algorithm an unbiased estimator of the gradient V Vp)=
Algorithm 6 SamSA: Sample 8, for SPG (Algorithm 8.1 in Ding al. (2021))
1: Input: p, 0,0,7
2: Draw H ~ Geom(1 -17)
>> Geom(1 - 7) geometric distribution with parameter 1
Draw 5g ~ p. 00 (-(-)5))
4: for h = -1, H do
5:
Draw 8h+1 P(-|sh, an) ( ap-1 ~ To, (118811)1)
6: end for
7: Output: 8H, 0
Algorithm 7 EstEntQ: Unbiased Estimation of Maxlit Q (Algorithm 8.2 in Ding et al. (2021))
1: Input: a, 0,,,0,,,00
2: Initialize 80 f- 8, +- a, Q + r(50: no)
3: Draw H Geom(1 7)
4: for h = 0......,,1
do
5: 8h+1 P( 0n) aht1 To =0(188++1)
6:
Q Q + + /2 (8 ux+1) - a log To (a
7: end for
8: Output: Q
27

---

Under review as a conference paper at ICLR 2023
APPENDIX E EXPERIMENTAL DETAILS
We use the SAC implementation from https://github. com/ikostrikov/
jaxrl (Kostrikov, 2021) for all our experiments in the paper. The exception is the goal reaching task with automated curriculum generation, where we use our customized DDPG implementation that is highly inspired from the same codebase.
E.1 GOAL REACHING WITH AN ORACLE CURRICULUM
For our antmaze umaze experiments with oracle curriculum, we use sparse reward function where the reward is when the distance D between the ant and the goal is greater than 0.5 and r cxp(-5D) when the distance smaller than equal 0.5. The performance threshold is set to be R 200. Exceeding such threshold means that the ant stays on top of the desired location for least 200 out of 500 steps, where 500 is the maximum episode length the antmaze sammze--maa ze environment. We use the average return of the last 10 episodes and compare it to the performance threshold R. For both of the SAC agents, we use the same set of hyperparameters shown Table See Algorithm 8, for more detailed pseudocode.
Initial Temperature
1.0
Target Update Rate
update rate of target networks
0.005
Learning Rate
learning rate for the Adam optimizer
0.0003
Discount Factor
0.99
Batch Size
256
Warmup Period
number of steps of initial random exploration (random actions)
10000
Network Size
(256, 256)
Table 3 Hyperparameters used for the SAC algorithm (Haarnjja et al., 2018)
28

---

Under review as a conference paper at ICLR 2023
Algorithm 8 Practical Implementation of ROLLIN
{u input curriculum, pt initial state distribution, near-optimal threshold, 8:: roll-in ratio, discount factor
Input:
2: Initialize < 0, Dexp +- k +- 0, and two off off-policy RL agents Tmain and Texp-
3: for each environment step do
4:
if episodd terminating or beginning of training then
5:
if average return of the last 10 episodes under context We is greater than R then +- k + 1, Dexp + Re-initialize the exploration agent Texp
6:
7:
8:
end
9:
Start new episode under context wg with so ~ p.
10:
if k > 0 and with probability of 3 then
enable Rollin for the current episode.
12:
else
disable Rollin for the current episode.
14:
end if
end
16:
if Rollin is enabled for the current episode then
17:
Rollin stopped for the current episode then at Texp (ag 88, wk)
18:
19:
else
20:
at e=-aain (ee wk-11
21:
with probability of 1 - 7,, stop Rollin for the current episode
22:
end if
23:
else
at Tmain (at (8c, wk)
25:
end if
take action at in the environment and receives 8++1 and ri = rwn (8e, ae)
27:
add (st, ar St-1. ) in replay buffer D
28:
if Rollin is disabled for the current episode then
29:
update maain using D.
30:
end
31:
if
Texp used to produce a, then add (St, Ot, 8141, r2) in replay buffer Doxp update Texp using Doxp-
32:
33:
34:
end if
+ +++1
36:
end for
Output: Tinai
E.2 GOAL REACHING WITH AN AUTOMATICALLY GENERATED CURRICULUM
For our antmazer uma ze experiments with automatically generated curriculum, we use a sparse reward function where the reward is -1 when the distance between the ant and the goal is greater than 1.0 (this is different from the oracle curriculum setting where the threshold is 0.5), and 0 otherwise. We closely follow the DDPG setup in the MEGA paper (Pitis et al., 2020) and
the rfaab relabeling with a ratio of (0.1, 0.4, 0, 0.3, 0) We also use the go-explore style exploration where the epsilon- geriooy-greeyyyoo ratio is increased from 0.1 0.11 when the goal is reached (whenever a reward of zero is received). The maximum episode length is 500 and we estimate the success rate of the agent by rolling out the policy times with different pairs of random initial starting locations (a square region centered around (0.0, 0.0) with length 2.0) and goals (a square region centered around (0.0, -8.0) with length 2.0). See Figure for a visualization three-layer network (512. 5 512, 512) with layer normalization for both the actor and the critic. We also use target Q value clipping such that the target Q value that gets backed up is always within --00, 0] (the range of the achievable value with discount factor of 0.99 and binary reward (-1., 0.)). We did not use a dynamic normalization of the observation (which was used the original paper). Also, used tanh squash the output of the actor be between -1 and rather than using 7.2 action
successful trajectories
this task. We use
29

---

Under review as a conference paper at ICLR 2023
penalty done in the original paper. We include detailed description of how ROLLIN is applied on top of MEGA in Algorithm 9 with = 10 and the goal proposer from the MEGA paper with the default parameters. See Figure 4 for the results of ROLLIN top of MEGA over range of | values.
Baseline
B 0.75
B = 0.2
_ = 0.9
B = 0.5
B = 0.1
1.00
Success Rate
0.75
0.50
0.25
0.00
2
3
4
Env. Steps (x10%)
Figure 4 The success rate of goal reaching in antmaze-umaze with curriculum generated by MEGA with B = 0.1, 0.2, 0.5, 0.75, 0.9. The success rate is reported over 8 independent random seeds. 10 random trials for each seed are used to estimate the success rate.
5
-10
-10
-5
5
Figure 5 Visualization of the successful trajectories by taking the ant's 2-D location in the maze. The maze walls are colored in black. The circles represent the starting locations of the agent and the crosses with the same color represent their corresponding goal locations.
30

---

Under review as a conference paper at ICLR 2023
Algorithm 9 Practical Implementation of ROLLIN when combined with Automated Goal Curriculum
Generation
1: Input: initial state distribution, 3: roll-in ratio, discount factor goal proposer G, (c.g., MEGA (Pitis et al., 2020)), curriculum step interval N,
2: Initialize D t 0, Dexp f- 0. k t- 0, and two off- off-policy RL agents Tmain and Texp-
3: for each environment step do
4: if episode terminating or beginning of training then
5:
more than steps have been taken at curriculum step then
6:
à k + 1, D Pexp f- Re-initialize the exploration agent Texp: Tmain,k Tmain,k-1
7:
8:
end if
9:
Start a new episode with So ~ p. + O, get new goal 9 proposed by G. if k >> 0 and with probability of then
11:
enable Rollin for the current episode.
else
13:
disable Rollin for the current episode.
end if end if
15:
if Rollin is enabled for the current episode then
17:
if Rollin is stopped for the current episode then
18:
at ~ Texp (g 8t, (t,
19:
else
20:
a ~ Am (t.
21:
with probability of 1 7, stop Rollin for the current episode
22:
end
else
24:
0r T (t.lst,g)
25:
end if
26:
take action de in the environment and receives 81+1 add (st, at, St-1. g) in replay buffer DD if Rollin is disabled for the current episode then update Tmain using with goal relabeling. end if
28:
29:
30:
31:
if Texp was used to produce a, then
32:
add (st, de, 81+1, g) in replay buffer Dexp update Texp using Dexp with goal relabeling. end if
33:
34:
35:
++1
36:
end for
37:
Output: Amain,k
E.3 NON GOAL REACHING
For the non goal reaching tasks in walker2d, hopper, humanoid, and ant experi- ments, the desired z-velocity range [Ax A/s 0.1)), the near-optimal threshold R(k), and the healthy reward all depend on the environments, The maximum episode length 1000. Details provided in Table 4.
@eearhhyrewardd
31

---

Under review as a conference paper at ICLR 2023
Weellhy_rewand
X
R(s)
original
high
low
500
hoppez
500
4500x
1.0
1.5
humanoid
1 2500 + 2500s
5.0
7.5
2.5
ant
6 500
4500x
1.0
1.5
0.25
Table 4 Learning progress K, average z-velocity, and average return at the 0.75 and 1.0 million environment steps in walker, hopper, humanoid, and ant. The average z-velocity and return estimated using the last 50k time steps. We pick found in Table 8, 9, 10 Appendix F.3. The standard error is computed over 8 random seeds.
for all experiments using ROLLIN, the results of using other Os can be
APPENDIX F ADDITIONAL LEARNING CURVES AND TABLES
F.1 GOAL REACHING
Baseline
0.75
3= 0.2
3 0.9
0.5
==0.1
A= 1/24
A=1/18
A= 1/12
Learning Progress (xc
Learning Progress (x)
Learning Progross (x)
0.50
0.50
0.50
0.25
0.25
0.00
0.00
0.00
0.0
1.0
2.0
2.5 3.0
0.0
1.0
1.5
2.0
2.5 3.0
0.0
0.5
1.0
1.5
2.5.3
Env.
Steps (x10*)
Env. Steps (x10*)
Env.
Steps (x10*)
Vanilla ROLLIN without geometric sampling
A=1/24
A=1/18
A=1/12
Learning Progress (
Learinng Progress (x)
1.0
1.0
Leariing Progress
0.5
0.5
0.5
0.0
0.0
0.0
0.5
1.0
2.0
2.5 3.0
0.0
0.5
2.1.1.2....55. 3.0
0.0
2.5.
Env. Steps (x10%
Env. Steps (x10%)
Env. Steps (x10*)
Vanilla ROLLIN with geometric sampling
Figure Vanilla Goal reaching. Accelerating learning on antma ze umaee with ROLLIN on an oracle curriculum in Figure 2. The confidence interval represents the standard error computed over random seeds.
32

---

Under review as a conference paper at ICLR 2023
Baseline
5 = 0.75
8= 0.2
B
== 0.1
A=1/24
A=1/18
A=1/12
Learning Progress (x0
Learning Progress (x)
1.0
1.0
1.0
Learning Progress
0.5
0.5
0.5
0.0
0.0
0.0
0.0
1.015.2.0
3.0
0.0
1.0
1.5
0.0
1.0 2.005.00
Env. Steps (x109)
Env. Steps (x10%)
Env. Steps (x109)
ROLLIN + relabeling without geometric sampling
1=1/24
A=1/18
1-122
Learning Progress (16
1.0
Learning Progress (x)
1.0
Learning Progress (x)
1.0
0.5
0.5
0.5
0.0
0.0
0.0
0.0
2..
0.00.5
2.0.2.
0.0 1.0 2.2022530
Env.
Steps (x10*)
Env. Steps (x10")
Env. Steps (x10")
ROLLIN + relabeling with geometric sampling
Figure Goal relabeling. Accelerating learning on antmaze-umaze with ROLLIN on an oracle curriculum in Figure 2. The confidence interval represents the standard error computed over 8 random seeds.
Baseline
8 0.75
8
0.2
0.9
0.5
8 0.1
A A= 1/14
A=1/18
A=1/12
Learning Progress (x)
Learuing Progres (x)
Learning Progress (x)
0.4
0.4
0.50
0.2
0.2
0.25
0.00
0.0
0.5
1.5 1.5 2.0 2.0
0.0
1.0
2.0
2.5 3.0
0.0 1.0 1.5.022.330
Env. Steps (x10%)
Eny. Steps (x10%)
Env. Steps (x10*)
ROLLIN Go-Explore without geometric sampling
A A 1/24
A= 18
1/12
Learring Progress (x)
1.0
Learning Progress (x)
Learning Progress (x)
0.4
0.5
0.5
0.2
0.0
0.0
0.0
0.0
2.5.
0.0
1.0 2.5.0.25.55
0.5
1.0
2.2.0....
Env. Steps (x10*)
Env. Steps (x10*)
Env. Steps (x10%)
ROLLIN + Go-Explore with geometric sampling
Figure Go-Explore (exploration noise 0.1). Accelerating learning on antmaze-umaza with ROLLIN on an oracle curriculum in Figure 2. The confidence interval represents the standard error computed over 8 random seeds.
33

---

Under review as a conference paper at ICLR 2023
Baseline
8 = 0.75
== 0.2
0.9
0.1
A 3-1/21
A A=1/18
- 1/12
Learingg Progress (e)
Learuing Progress (x)
Tearning Progress (K)
0.4
0.50
0.2
0.2
0.25
0.00
0.0
0.0
0.0 0.5
1.5
1.0
1.5
2.5 3.0
0.0
0.5
1.5
2.0 2.5 3.0
0.5
1.0 1.5 2.500.2550
Env.
Steps (x10%)
Fny. Steps (x10*)
Fny.
Steps (x10*)
ROLLIN Go-Explore without geometric sampling
1-1/24
A=1/18
A= 1/12
Learning Progress (x)
Learning Progress (x)
Learning Progress (x)
1.0
0.4
0.5
0.5
0.2
0.0
0.0
0.0
0.0 1.0
0.00.5 1.0 2.5.02....0
0.0
1.0 1.2.022.55
Euv. Steps (x109)
Env. Steps (x10%)
Env. Steps (x109)
ROLLIN + Go-Explore with geometric sampling
Figure Go-Explore (exploration noise 0.25). Accelerating learning on antmaze umaze with ROLLIN on an oracle curriculum in Figure 2. The confidence interval represents the standard error computed over 8 random seeds.
Bascline
0.75
0.2
B = 0.9
0 = 0.5
== 0.1
A=1/24
A= 1/18
1=12
Learning Progress (16)
Leariing Progress (x))
Learning Progress (x)
0.4
0.4
0.5
0.2
0.2
0.0
0.0
0.0
0.0 1.0 2.0 3.0
0.0
1.0 1.5
2.5 3.0
0.0
1.0 1.5 2.5
Env. Steps (x10%)
Env. Steps (x10%)
Env. Steps (x10*)
ROLLIN Go-Explore without geometric sampling
1/24
A= 18
1/12
Learring Progress (x)
Learning Progress (x)
Learning Progress (x)
1.0
0.4
0.5
0.5
0.2
0.0
0.0
0.0
0.0
2.0...200..53
0.0
2.0
2.3
0.0
1.0
2.2.0....3
Env. Steps (x10*)
Env. Steps (x10*)
Env. Steps (x10%)
ROLLIN + Go-Explore with geometric sampling
Figure Go-Explore (exploration noise 0.5). Accelerating learning on antmaze-umav with ROLLIN on an oracle curriculum in Figure 2. The confidence interval represents the standard error computed over 8 random seeds.
34

---

Under review as a conference paper at ICLR 2023
Geo
A
Baseline
8= 0.1
Ø 0.2
8 0.5
B = 0.75
8 0.9
1/24
0.40
0.02
0.49 +: 0.04
0.51 #: 0.05
0.57 +: 0.04
0.55 +: 0.02
0.49 0.02
1/18
0.36 1 0.01
0.39 1: 0.01
0.46 1: 0.01
0.54 + 0.03
0.46 1: 0.03
0.50 1: 0.02
1/12
0.36
0.44 + 0.01
0.41 4:
0.47 + 0.02
0.05
0.56 0.02
1/24
0.82 1 0.08
0.92 1 0.02
0.95 0.02
0.02
0.88 +: 0.01
0.81 1: 0.01
0.70 1 0.02
1/18
0.68
0.07
0.74 4 0.07
0.76 4: 0.06
0.78 + 0.03
0.75
1: 0.02
0.72 0.02
1/12
0.38
= 0.03
0.50 = 0.04
0.55 = 0.04
0.64 = 0.06
0.69 = 0.06
0.67 + 0.03
Table Vanilla Goal reaching. Learning progress k at 3 million environment steps with varying 3 and curriculum step size A of vanilla goal reaching task. Geo indicates the usage of geometric sampling. Baseline corresponds to O. where no ROLLIN is used. The standard error computed over 8 random seeds. We highlight the values that are larger than the baseline (B 0) purple, and the largest value bold font.
A
0.1
0.2
0.5
0.75
3 0.9
1/24
0.80
0.91 0.03
0.85 0.04
0.86 0.02
0.02
1/18
0.76 = 0.03
0.81 0.01
0.85 = 0.04
0.001
0.81 0.03
0.68 0.04
1/12
0.66 + 0.04
0.74 0.01
0.73 0.03
0.66 0.06
0.67 0.05
0.69 0.03
1/24
0.76 = 0.02
0.78 0.01
0.78 0.08
0.82 0.002
0.87 + 0.02
0.83 0.02
1/18
0.72
0.76
0.76 0.02
0.85 0.03
1/12
0.72
1 0.03
0.73 + 0.00
0.73 + 0.00
0.73 + 0.03
0.69 1 0.04
0.66 0.04
Table 6 Goal relabeling. All other settings are the same as Table 5.
EN Gec
A
8=
8=
A= 0.2
0
[ 0.75
0 8= 0.9
x
1/24
0.37
0.00
0.52
0.54 + 0.06
0.51 0.06
0.39 + 0.02
0.37 0.01
0.1
x
1/18
0.38
0.00
0.37 0: 0.01
0.36 0.01
0.38 + 0.01
0.38 = 0.01
0.35 0 0.00
1/12
0.001
3.8:0.000
0.00
0.00
0.1
1/24
0.82
0.007
0.95 0.02
0.91 0.02
0.92 0.02
0.71 0.02
0.45 0.01
0,1
1/18
0.57
0.09
0.71 + 0.08
0.63 = 0.07
0.63 - 0.07
0.62 - 0.02
0.43 + 0.01
0.1
1/12
0.42
1 0.03
0.43 0.02
0.38 1 0.04
0.49 + 0.04
0.45 1 0.02
0.39 1: 0.01
0.25
x
1/24
0.38
0.02
0.49 - 0.06 0.066
0.48 + 0.05
0.55 + 0.07
0.43 + 0.04
0.40 1:0::00 0.02
1/18
0.39
0.25
1/12
0.36
0.00
0.36 + 0.00
0.36 + 0.00
0.36 + 0.00
0.36 + 0.00
0.36 + 0.00
0.25
1/24
0.82 0.10
0.97 1: 0.02
1.00 + 0.00
0.02 0.94 0.02
0.77 1: 0.02
0.02 0.41 0.01
0.49 0.00
0.25
1/18
0.61
0.70
0.79 + 0.07
0.61
0.63 = 0.03
1/12
0.001
0.47
0.04
0.40
X 1/24
0.49
0.49
0.60 0.08
0.66 0.08
0.61 + 0.08
0.65 + 0.06
0.46 0.01
0.5 X
1/18
0.36
0.01
0.41 t 0.02
0.39 0.01
0.38 + 0.01
0.37 + 0.01
0.35 : 0.00
X
1/12
0.36
0.00
0.38 0.01
0.38 + 0.01
0.36 1: 0.00
0.38 + 0.01
0.36 1: 0.00
0.5
/
1/24
0.92 0.08
1.00 + 0.00
1.00 + 0.00
1.00 + 0.00
0.87 + 0.03
0.55 +
0.03 0.03
1/18
0.09
0.08
0.08
1 0.04
0.02
0.5
/
1/12
0.41
0.00
0.44 4.44 0.04
0.43 + 0.03
0.50 + 0.04
0.47 - 0.03
0.39 + 0.04
Table 7 Go-Explore with different exploration noise. EN represents the multiplier for the Gaussian exploration noise. All other settings the same Table 5.
F.2
GOAL REACHING WITH A LEARNED CURRICULUM
Baseline
0= 8 0.5
0 = 0.75
8 0.9
8 0.1
1.00
Sucosss Rate
0.75
0.50
0.25
0.00
2
4
5
Enc.
Steps (x10')
Figure 11 Performance of ROLLIN combined with MEGA (Pitis et al., 2020) on antmaze -arimazze-mzzeee with different choices of B.
35

---

Under review as a conference paper at ICLR 2023
F.3
NON GOAL REACHING TASKS
Baseline
0.1
walker
walker
walker
1.0
Learning Progress
4000
velocitty
return
0.5
2000
1.0
0.0
0.0
0.5
Env. Steps (x109)
Env. Steps (x10%
Env. Steps (x109)
hopper
hopper
hopper
Learning Progress (x)
velocity
2
roturn
0.5
2000
0.0
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Env. Steps (x 10%)
Env. Steps (x10²)
Env. Steps (x10*)
humanoid
humanoid
humanoid
Learning Progress (x)
4000
0.5
velocity
0.5
Teburn
2000
0.0
1.0
0.0
0.5
0.0
0.5
Env. Steps (x109)
Env. Steps (x109)
Env. Steps (x10%)
ant
ant
ant
Leaming Progress (fc)
1.0
5.0
4000
velocity
urr
0.5
2.5
2000
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Env. Steps (x108)
Env. Steps (x109)
Env. Steps (x10%)
Figure 12 Accelerating learning on several non goal-reaching tasks. The confidence interval represents the standard error computed over 8 random seeds, for 3 0.1.
36

---

Under review as a conference paper at ICLR 2023
Bnv.
Step
0 ==00
0 = 0.1
= 0.2
B = 0.5
0 0.75
ker
0.75m
0.89
0.03
0.90 +: 0.02
0.85 0.03
0.88 0.04
0.85 +: 0,04
Im
0.92 +: 0.03
0.94 t 0.03
0.90 = 0.01
0.92 1 0.04
0.92 1: 0.03
hoppe
0.75m
0.86 + 0.02
0.85 0.02
0.88 + 0.02
0.81 + 0.03
0.79 + 0,01
1m
0.88 1: 0.01
0.89 0.00
0.89 + 0.03
0.82 1 0.02
0.81 +: 0.02
idd
0.75m
0.54 + 0.01
0.61 0.05
0.56 = 0.03
0.67 + 0.03
0.61 + 0.05
0.03
0.06
0.71 0.06
ant
0.75m
0.93
0.02
0.99
0.01
0.79 = 0.09
0.79 + 0.06
0.61 + 0.07
1m
1.00
0.00
1.00 +: 0.00
0.83 0.08
0.86 + 0.06
0.71 + 0.07
Table 8 Learning progress к at 0.75 and 1.0 million environment steps with varying B of goal reaching tasks. Baseline corresponds to where ROLLIN is used. The standard error is computed over random seeds. We highlight the values that are larger than the haseline (3 0) purple, and the largest value in bold font.
kav.
Step
9=
0.1
0.2
8 0.5
B 0.75
walker
0.75m
3.20
0.40
3.48
0.29
2.99 0.26
3.08 0.30
3.25 + 0.38
Im
3.60
0.27
3.62 0 0.26
3.09 0.28
3.48 + 0.27
3.14 + 0.34
hoppar
0.75m Im
2.54 0.17
0.19
0.17
2.61 +: 0.17
0.16 2.58
2.65 + 0.15
2.65 + 0.17
2.39 + 0.18
2.52 0.19
humanaid
0.75m
0.32
0.05
0.39 0.06
0.34 0.05
0.41 + 0.07
0.41 + 0.09
Im
0.39
= 0.05
0.46 + 0.09
0.41 + 0.05
0.41 + 0.06
0.49 + 0.10
ant
0.75m
4.30
0.37
4.50 + 0.30
3.55 1: 0.54
3.47 1. 0.57
3.21 1: 0.50
0.51
4.66
0.30
3.93 0.45
3.99
0.48
3.50 +: 0.49
Table 9 Average r-direction velocity of the last 50k time steps, at 0.75 and 1.0 million environment steps with varying 3 of non goal reaching tasks. Baseline corresponds to B = O, where no ROLLIN is used. The standard error is computed over 8 random seeds. We highlight the values that are larger than the baseline (5 = 0) in purple, and the largest value in bold font.
Step
8
A =
8 0.5
B 0.75
walker
0.75m
3838.2 + 234.0
3831.7
133.3
3550.3 + 115.0
3772.6 + 158.6
3667.3 340.4
1m
41332.3 +: 224.3
4128.8 1 159.6
3680.5 à 135.6
4028.8 8 4 164.2
3896.4 205.4
hopper
0.75m
3325.8 = 100.6
3339.7 + 151.4
3219.4 + 89.4
3008.6 4: 133.4
3219.2 142.3
1m
3386.2 1: 124.7
3421.9 + 109.8
3262.3 1 98.1
3170.7 1 180.6
3394.. 126.5
huranoid
0.75m
3007.6 + 176.6
3151.0 + 171.6
2848.0 + 176.7
3065.22
226.4
3224.7 +: 266.2
Im
3017.2 #: 169.0
3173.6 4 238.3
2935.8 +: 181.1
2905.5
125.9
3290.7 1: 275.9
ant
0.75mm
3151.0 1: 224.4
4242. 1.422.1::0 + 89.5
3259.8 1: 434.1
3196.9
434.6
2739. 9 360.7
mm
4248.5 88.6
4473.0 + 102.2
3683.1 1 345.0
3708.7 290.5
$250. 1 316.2
Table 10 Average return of the last 50k time steps, the 0.75 and 1.0 million environment steps with varying of non goal reaching tasks. Baseline corresponds computed over 8 random seeds. We highlight the values that are larger than the baseline (/3 = 0) purple, and the largest value in bold font.
where no ROLLIN is used. The standard error
APPENDIX G ADDITIONAL DISCUSSIONS ON THE RELATED WORK
Liu et al. (2022); Bassich et al. (2020) also consider RL tasks with curriculum. From theoretical perspective, we demonstrate the sample complexity gain in the stochastic policy gradient method using non-asymptotic statistical analysis techniques, whereas Liu et al. (2022); Bassich et al. (2020) do not study policy gradient methods.
From an empirical perspective, the non goal-reaching Mujoco locomotion domain is related to the continuous robot evolution model considered by Liu et al. (2022). This prior work uses a context to characterize the transition dynamics of the robot, while assume the same transition dynamics for all tasks and our context only determines the reward function. The major empirical difference between our work and Liu al. (2022) lies in the algorithm. ROLLIN facilitates learning by rolling
37

---

Under review as a conference paper at ICLR 2023
in "near-optimal" policy learned from the previous context for learning the next context, which enables the agent to start from an initialization that is "close" to the optimal policy specified by the current context, while Liu al. (2022) stores all transitions of different robot parameters into one replay buffer, and uses the replay buffer to learn a single policy.
Our goal-reaching tasks in the antma - environment also appears in Bassich et al. (2020). However, our work studies a problem that is different to Bassich et al. (2020). In particular, Bassich al. (2020) considers curriculum learning for RL by proposing progression function that generates the context for each task, while our method tackles the orthogonal problem of using given curriculum accelerate RL. Our method agnostic to where this curriculum comes from, and it could be produced by the designer (Section 6.1) or another algorithm (Section 6.2).
APPENDIX H ADDITIONAL DISCUSSIONS ON THE LIMITATIONS
Our theoretical analysis in the tabular case inspires practical implementation of ROLLIN which demonstrates empirical success in various domains. However, we shall clarify that theoretical analysis in the tabular case is still limited for understanding the empirical success of RL practice, as the state spaces in many RL applications are either enormous continuous. One potential future direction to extend the initial distribution update procedure (I) from tabular analysis to feature space analysis.
On the practical side, although ROLLIN generally demonstrates empirical benefits with a choice of 3 = 0.1 or 3 = 0.2, one still needs to fine tune the 3 for achieving the best performance in general. Hence, another future direction on the empirical side could be implementing the an auto- auto-tunnig version of the ROLLIN parameter B.
38


Reviews:
[{"review_id": "zzdOWS5LxF", "reviewer": "Reviewer_cM8j", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a theoretically-motivated framework that reformulates a single-task RL problem as a multi-task RL problem defined by a curriculum for computationally-efficient policy learning. Their framework and other baseline are demonstrated in a goal-reaching task with a Mujoco dynamical system. The results show improved learning performance compared to traditional methods. ", "summary_of_the_review": "Overall, the presented approach is novel with theoretical support. Experiments demonstrate better performance than the prior methods. However, test cases with different dynamical systems and investigation on the applicability of the proposed approach in speeding up learning across different environments can further strengthen the paper.\n", "strength_and_weaknesses": "Strengths:\n- A novel approach for efficient RL policy learning.\n- Algothrim seems sound with theoretical backing.\n- Experiments demonstrate that the proposed approach leads to better performance. \n\nWeaknesses\n- The experiment section is relatively weak. It is unclear how the proposed approach will scale to other Mujoco robots, such as Swimmer and Humanoid. \n- Is the idea of a warm start to policy learning also applicable across different environments? For instance, using parameters from u-maze to some other maze environment could aid in investigating that.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PkDsL4PqX5", "reviewer": "Reviewer_haUz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes ROLLIN, and algorithm to reduce the complexity of learning an optimal policy in a sequence of contextual MDPs, i.e. a set of MDPs sharing everything but the reward function in this work. The main idea of the paper is that if the two contexts are close enough, we can modify the initial state distribution in a way to reduce the total computational complexity of a policy gradient method.\n", "summary_of_the_review": "My current recommendation 'below acceptance' is mostly caused by the empirical evaluation protocol weaknesses. I am ready to increase the score if my concerns from the 'weaknesses' are addressed.", "strength_and_weaknesses": "\n### Strengths\n\n* The paper proposes a neat way to reduce sample efficiency in the setting providing the theoretical proof (I did not check all the derivations in the appendix).\n\n### Weaknesses\n\n* The paper chooses a non-standard way of evaluating the algorithm, i.e., providing the learning progress value instead of the policy returns. As far as I understand, we have an ordered set of MDPs, and we are eventually interested in solving the target one. I believe, the chosen evaluation protocol favours the proposed method, and I would like to see the comparison: curriculum vs baseline on the target task only.\n* The proposed algorithm requires knowing the optimal parameters for the first MDP in the set. If I get it correctly, if we remove this quite limiting assumption, we will get the exponential complexity again, though it will still be reduced by the consequent application of ROLLIN.\n* The paper misses important connection to the related work, which is more empirical, but very similar in spirit:\n    * REvolveR: Continuous Evolutionary Models for Robot-to-robot Policy Transfer, ICML 2022\n    * Curriculum Learning with a Progression Function, Bassich et al.\nIn addition, I think the paper misleads the reader by its title. Usually, by multitask RL, we mean a single model being able to solve multiple tasks at once, whereas this paper looks at the linear curriculum with a single target task.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7jzicDBx7AF", "reviewer": "Reviewer_5mDQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Under the assumptions of 1) The reward function is lipschitz continuous w.r.t the context and 2) maximum distance between two contexts, this paper proposes an algorithm \"Rollin\" to learn an optimal policy for the final context and show that it can be achieved in polynomial iteration complexity instead of exponential. The empirical results were shown on a toy domain.", "summary_of_the_review": "As listed in the weaknesses, this paper makes very strong assumptions that makes the proposed algorithm impractical to use. The authors should come up with an algorithm to automatically generate the curriculum of contexts. In its current form, I recommend the paper as a weak reject. ", "strength_and_weaknesses": "Strengths:\nThe paper provides novel polynomial time iteration complexity and sample complexity bounds for learning a single-task given a curriculum of contexts. \n\nWeaknesses:\n1) Throughout the paper, the authors carry the tone of learning a single-task policy by recasting it as a multi-task problem. (e.g, \"In this work, we provide a theoretical framework that reformulates a single-task RL problem as a multi-task RL problem defined by a curriculum..\" in the abstract) But, the proposed algorithm \"Rollin\" assumes that the curriculum (contexts) are given as input. \n2) The assumption of having a curriculum that satisfies the assumptions 3.1 and 3.2 is a very strong assumption. In most algorithms involving goal-conditioned RL or curriculum learning, such a curriculum is not provided. \n3) They also assume that the optimal policy for the initial context is known.\n4) The empirical results are shown on a very simple toy domain.\n5) There are multiple grammatical errors / typos throughout the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject