Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning
Authors: Sasha Salter
Kristian Hartikainen
Walter Goodwin
Ingmar Posner
Venue: ICLR 2023 poster (2023)
Keywords: Skills
Transfer Learning
Reinforcement Learning

Abstract: The ability to discover behaviours from past experience and transfer them to new tasks is a hallmark of intelligent agents acting sample-efficiently in the real world. Equipping embodied reinforcement learners with the same ability may be crucial for their successful deployment in robotics. While hierarchical and KL-regularized reinforcement learning individually hold promise here, arguably a hybrid approach could combine their respective benefits. Key to these fields is the use of information asymmetry across architectural modules to bias which skills are learnt. While asymmetry choice has a large influence on transferability, existing methods base their choice primarily on intuition in a domain-independent, potentially sub-optimal, manner. In this paper, we theoretically and empirically show the crucial expressivity-transferability trade-off of skills across sequential tasks, controlled by information asymmetry. Given this insight, we introduce Attentive Priors for Expressive and Transferable Skills (APES), a hierarchical KL-regularized method, heavily benefiting from both priors and hierarchy. Unlike existing approaches, APES automates the choice of asymmetry by learning it in a data-driven, domain-dependent, way based on our expressivity-transferability theorems. Experiments over complex transfer domains of varying levels of extrapolation and sparsity, such as robot block stacking, demonstrate the criticality of the correct asymmetric choice, with APES drastically outperforming previous methods.

Published as a conference paper at ICLR 2023
PRIORS, HIERARCHY, AND INFORMATION ASYMMETRY FOR SKILL TRANSFER IN REINFORCEMENT LEARNING
Sasha Salter; Kristian Hartikainen, Walter Goodwin, Ingmar Posner University of Oxford (sasha, kristian, walter, hipoorooooooooooorrootticc institute
ABSTRACT
The ability to discover behaviours from past experience and transfer them to new tasks is hallmark of intelligent agents acting sample-efficiently in the real world. Equipping embodied reinforcement learners with the same ability may be crucial for their successful deployment in robotics. While hierarchical and KL-regularized reinforcement learning individually hold promise here, arguably a hybrid approach could combine their respective benefits. Key to these fields the use of infor- mation asymmetry across architectural modules to bias which skills are learnt. While asymmetry choice has large influence on transferability, existing methods base their choice primarily on intuition in domain iamain-indeeendent, potentially sub- optimal, manner. In this paper, we theoretically and empirically show the crucial trade-off of skills across sequential tasks, controlled by information asymmetry. Given this insight, we introduce Attentive Priors for Expressive and Transferable Skills (APES), hierarchical KL-regularized method, heavily benefiting from both priors and hierarchy. Unlike existing approaches, APES automates the choice of asymmetry by learning it in a data-driven, domain- dependent, way based on our over complex transfer domains of varying levels of extrapolation and sparsity, such as robot block stacking, demonstrate the criticality of the correct asymmetric choice, with APES drastically outperforming previous methods.
theorems. Experiments
INTRODUCTION
While Reinforcement Learning (RI) algorithms recently achieved impressive feats across range of domains (Silver et al. [2077] Mnih et al., 2015 Lillicap et al., 2015), they remain sample inefficient (Abdolmaleki et al. 2018 Haarnoja et al. 2018b and are therefore of limited use for real- world robotics applications. Intelligent agents during their lifetime discover and reuse skills at multiple levels of behavioural and temporal abstraction to efficiently tackle new situations. For example, in manipulation domains, beneficial abstractions could include low-level instantaneous motor primitives as well higher-level object manipulation strategies. Endowing lifelong learning RL agents (Parisi) et al. 2019) with a similar ability could be vital towards attaining comparable sample efficiency.
To this end, two paradigms have recently been introduced. KL-regularized RL TTeh et al. 2017] Galashov et al., 2019) presents an intuitive approach for automating skill reuse in multi-task learning. By regularizing policy behaviour against a learnt task-agnostic prior, common behaviours across tasks are distilled into the prior, which encourages their reuse. Concurrently, hicrarchical RL also enables
skill discovery (Wulfmeier et al., 2019 Merel et al., 2020 Hausman et al., 2018 Haarnjja al.. 2018a Wulfmeier et al. (020)) by considering a two-level hierarchy in which the high-level policy is fask-conditioned, whilst the low-level remains task-agnostic. The lower level of the hierarchy therefore also discovers skills that are transferable across tasks. Both hierarchy and priors offer their own skill abstraction. However, when combined, hicrarchical KL-regularized RL can discover multiple abstractions. Whilst prior methods attempted this (Tirumala et al.) 2009 2020 Liu et al.) 2022 Goyal et al. 2019). the transfer benefits from learning both abstractions varied drastically, with approaches like Tirumala et al. (219) unable to yield performance gains.
In fact, successful transfer of the aforementioned hierarchical and KL-regularized methods critically depends on the correct choice of information asymmetry (IA). IA more generally refers to an asymmetric masking of information across architectural modules. This masking forces independence to, and ideally generalisation across, the masked dimensions (Galashov al., 2019). For example, for self-driving cars, by conditioning the prior only on proprioceptive information it discovers skills
"Equal contribution. Correspondence to: sasha... sal ter@hotma com.

---

Published as a conference paper at ICLR 2023
independent to, and shared across, global coordinate frames. For manipulation, by not conditioning on certain object information, such as shape or weight, the robot learns generalisable grasping independent to these factors. Therefore, IA biases learnt behaviours and how they transfer across environments. Previous works predefined their IAs, which were primarily chosen on intuition and independent of domain. In addition, previously explored asymmetries were narrow (Table D. which if sub-optimal, limit transfer benefits. We demonstrate that this indeed is the case for many methods on our domains (Galashov et al. 2019; Bagatella et al. 2022 Tirumala et al. 2019: 2020 Pertsch et al.. 2021 Wulfmeier et al., 2019). A more systematic, theoretically and data driven, domain dependent, approach for choosing IA is thus required to maximally benefit from skills for transfer learning.
In this paper, we employ hierarchical KL-regularized RL to effectively transfer skills across sequential tasks. We begin by theoretically and empirically showing the crucial trade-off, controlled by choice of IA, of skills across sequential tasks for hierarchical KI.-regularized RL. Our expressiityrransserabiltyy theorems state that conditioning skill modules on too little or too much information, such as the current observation or entire history, can both be detrimental for transfer, due to the discovery of skills that are either too general (e.g. motor primitives) or too specialised (e.g. non- transsfaeeeeee task-level). We demonstrate this by ablating over a wide range of asymmetries between the hierarchical policy and prior. We show the inefficiencies of previous methods that choose highly sub-optimal IAs for our domains, drastically limiting transfer performance. Given this insight, we introduce APES, 'Attentive Priors for Expressive and Transferable Skills' as a method that forgoes user intuition and automates the choice of IA in a data driven, domain theorems to learn the choice of
dependent, manner. APES builds on our expressivity-tramffrubbb asymmetry between policy and prior. Specifically, APES conditions the prior on the entire history, allowing for expressive skills to be discovered, and learns a low-entropic attention attention-mask over the input, paying attention only where necessary, to minimise covariate shift and improve transferability across domains. Experiments over domains of varying levels of sparsity and extrapolation, including complex robot block stacking one, demonstrate APES' consistent superior performance over existing methods, whilst automating IA choice and by by-passing arduous IA sweeps. Further ablations show the importance of combining hierarchy and priors for discovering expressive multi-modal behaviours.
2 SKILL TRANSFER IN REINFORCEMENT LEARNING
We consider multi-task reinforcement learning in Partially Observable Markov Decision Processes (POMDPs), defined by Mj- = (S, A, A.r P. ph: 7), with tasks is sampled from p(K)). S, A, X denote observation, action, and history spaces. pxx'|x, xx a) X' x X x A Rgo is the dynamics model. We denote the history of observations s E S. actions a E A up to timestep I as x, = (So, ag $1, ................................ st). Reward function TA X X Ax + R is history-, action- and task-dependent.
2.1 HIERARCHICAL KL-REGULARIZED REINFORCEMENT LEARNING
The typical multi-task KI Regreaalrr RL objective (Tooroov 2007 Kappen et al. 2012 Rawlik et al. 2012, Schulman et al. 2017) takes the form:
L( mo) Eyapp (7), kn)p(())
(x((x),)) a.) - oo (m(a|,, k) || mo(a(x;))
where 7 is the discount factor and og weighs the individual objective terms. x and To denote the task-conditioned policy and task-agnostic prior respectively. The expectation taken over tasks and trajectories 7 from policy π and initial observation distribution p (so). (i.e. (7)).. Summation over t occurs across all episodic timesteps. When optimised with respect π,, this objective can be viewed as a off between maximising rewards whilst remaining close to trajectories produced by Ta- When To is learnt, it can learn shared behaviours across tasks and bias multi-task exploration (Teh) et al. 2017). We consider the sequential learning paradigm, where skills are learnt from past tasks, Psource (K), and leveraged while attempting the transfer set of tasks, Ptrans(K)) (K).
While KL-regularized RI. has achieved success across various settings (Abdolmaleki et al.) 2018
Teh et al. 2017 Pertsch et al.) 2020 Haarnoja et al. 2018a), recently Tirumala et al. (2019) pro- posed a hierarchical extension where policy π and prior тo are augmented with latent variables, (aa 2 z/x, k) = π " (z/x, k)) (a|z, iz x) and To z x) = #! (z)x)=f' (a x). where subscripts H and L denote the higher and lower hierarchical levels. This strueture encourages the shared low-level policy (Tr = Tt') ) to discover agnkstti behavioural primitives, whilst the high-level discovers higher-level task relevant skills. By not conditioning the high-level prior on task-id, Tirumala et al.
2

---

Published as a conference paper at ICLR 2023
(2119) encourage the reuse of common high-level abstractions across tasks. They also propose the following upper bound for approximating the KL-divergence between hierarchical policy and prior:
DEL (i(aa)) |I To (a(a)))) & DKL (M (z|x)
||
# (z]x)) + E [DKL ( (a|x,, z) || मo (alx, 2))
We omit task conditioning and declaring shared modules to show this bound is agnostic to this. Table 1: Previously Explored IAs
2.2 INFORMATION ASYMMETRY
Information Asymmetry (IA) is key component in both of the aforementioned approaches. promoting the discov- ery of behaviours that generalise. IA can be understood the masking of information accessible by certain modules. Not conditioning on specific environment aspects forces independence and generalisation across them (Galashov et al. 2019). In the context of (hicrarchical)
Paper
impur
Iirumala
Dy Ci 2l
Patte ar
2001 211 Ajsy at
201
3020
Iirumala a1
2019
Goyal
KL-regularized RL, the explored asymmetries between the (high-level) policy, 7(H), and prior, m
(H)
have been narrow (Tirumala et al. [2019] 2020 Liu et al.] 2022, Pertsch et al. 2020|2022) Rao et al.)
2021; Ajay et al., 20200] Goyal al., 2019). Concurrent with our research, Bagatella et al. (2022)
published work exploring a wider range of asymmetries, closer to those we explore. We summarise
explored asymmetries in Table (with ac-nct representing action history up to n steps the past).
Choice of information conditioning heavily influences which skills can be uncovered and how well they transfer. For example, Pertsch et al.] (2020) discover inservatioo-dependern behaviours, such as navigating corridors in maze environments, yet are unable to learn hiistory-depedeen skills, such as never traversing the same corridor twice. In contrast, Liu et al. (022). by conditioning on history, are able to learn these behaviours. However, we will show, in many scenarios, naïvely conditioning on entire history can be detrimental for transfer, by discovering behaviours that do not generalise favourably across history instances, between tasks. We refer to this dilemma as the expressivity- transferability trade-off. Crucially, all previous works predefine the choice of asymmetry, based on the practitioner's intuition, that may be sub-optimal for skill transfer. By introducing theory behind the expressivityttrmnseerbbitt of skills, we present a simple data-driven method for automating the choice of IA, by learning it, yielding transfer benefits.
3 MODEL ARCHITECTURE AND THE TRADE-OFF
EXPRESSIVITY-TRAASFERAADI
To rigorously investigate the contribution of priors, hierarchy, and information asymmetry for skill transfer, it is important to isolate each individual mechanism while enabling the recovery of previous models of interest. To this end, we present the uni- fied architecture in Fig. [ which introduces information gating functions (IGFs) a means of decoupling IA from architec- ture. Each component has its own IGF, depicted with colored rectangle. Every module is fed all environment information Xx= (x, k) and distinctly chosen IGFs mask which part of the input each network has access to, thereby influencing which skills they learn. By presenting multiple priors, enable comparison with existing literature. With the right masking. one can recover previously investigated asymmetries (Tirumala et al. 2019; 2020: Pertsch et al. 2020 |Bagatella et al., 2022 Goyal al. 2019) ; explore additional ones, and also express purely hierarchical (Wulfmeier et al.
Figure I:
Hierarchical KL- regularized architecture. The hierarchical policy modules z and π L are regularized against their corresponding prior modules न.." and प. The inputs to each module are filtered by an information gating function (IGF), depicted with colored rectangles.
2019) and KL-regularized equivalents (Galashov et al. 2019; Haarnoja et al. 2018c).
3.1
THE INFORMATION ASYMMETRY
TRADE- OFF
While existing works investigating the role of IAs for skill transfer hierarchical KL-regularized RL have focused on multi- task learning (Galashov et al. 2019 we focus on the sequential task setting, in particular the prior's no ability to handle covariate shift. In contrast to multi-task learning. in the sequential setting, there exists abrupt distributional shifts, during training, for task p(AC) and trajectory pn (T) distributions. As such, it is important that the prior handles such distributional and covariate
Coocurrent with our research Bagatella et al. 2022 also investigated various LAs for sequential transfer.
3

---

Published as a conference paper at ICLR 2023
Ger Deeak
4CComizellaac
Figure 2: Environments. a) Mozridor The agent starts in the hallway and must traverse a given sequence of corridors. The agent completes a corridor by traversing to its depth and back. b) Stack: The agent must stack the cubes in given ordering over the light blue target pad.
shifts (see Theorem 3.1 for definition). For multi-task learning, trajectory shifts are gradual and skills are continually retrained over such shifts, alleviating transfer issues. In general, IA plays a crucial role, influencing the level of covariate shift encountered by the prior during learning:
Theorem 3.1. The more random variables a network depends on, the larger the covariate shift (input distributional shift, here represented by KL-divergence) encountered across sequential tasks, That is, for distributions p. q and inputs b. such that b = (bo, bi. b,.) and C b:
DEL (p(b) Il g(b)) 2 DEL (p(c) II q(()))
Proof. See Appendix B.I
In our case, p and q can be interpreted training Psource (.) and transfer Ptrans(:) (-) distributions over network inputs, such as history x, for high-level prior H. Intuitively, Theorem 3.1 states that the more variables you condition your network on, the less likely it will transfer due increased covariate shift encountered between source and transfer domains, thus promoting minimal information condi- tioning. For example, imagine conditioning the high-level prior on either the entire history Xo: or a subset of it X, -it E [0, t 1] (the subscript referring to the range of history values). According to Theorem 3.1 the covariate shift across sequential tasks will be smaller if we condition on a subset of the history, DEL (Psource (Xo:)) || Perans (Xo:))s(xo::)) 2 DKL (Psource (Xt (x+) Ptrans (xtrans(xe-m) (xt-nct) In- terestingly, covariate shift is upper-bounded by trajectory shift: DKL (Pryoergu (T) Pre.e.. (T)) Z DKL. sor (Ty) || Prtrans (Ty)) (using Theorem 3.1 with the right hand side representing covari- ate shift over network inputs Ty = ITF( filtered trajectories (e.g. Ty = X Ty C T), and Toource. Ttrans, source and transfer domain policies. It is therefore crucial, if possible, to minimise both trajectory and covariate shifts across domains, to benefit from previous skills. Nevertheless, the less information a prior is conditioned on, the less knowledge that can be distilled and transferred:
Theorem 3.2. The more random variables a network depends on, the greater its ability to distil knowledge in the expectation (output distributional shift between network and target distribution, here represented by the expected KL-divergence). That is. for target distribution p and network g with outputs a and possible inputs b, c, d, such that = (by, .... bn), C c C b, e E d +
Ey( [DLL (p(a(b) ll g(alc))] < DKL (p(a(b) || q(ald)).
Proof. See Appendix B.2
In this particular instance, p and q could be interpreted as policy т and prior To distributions, a as action a, b as history Xo.t, and d. subsets of the history (e.g. Xt-mat- Xt-mat= X4-m:1-m respec- tively, with n > m and m & n E (), with e denoting the set of variables in c but not d. Intuitively, Theorem 3.2 states in the expectation, conditioning on more information improves knowledge dis- tillation between policy and prior (c.g. Bro(xi-ne--m xx- [DRL (T(az) || mo (a, xx-n=))]] < DKL (a Xuxa) Tyyay (x-)m:))), with To (m=)))) the conditional distribution, induced by mo. of history subset X;- nm given xt-m):). Therefore, IA leads to
trade-off of skills (Theorems 3.1 (3.2). Interestingly, hierarchy does not influence covariate shift and hence does not hurt transfcrability, but it does increase network expressivity (c.g. of the prior), enabling the distillation and transfer of rich multi-modal behaviours present in the real-world.
4 APES: ATTENTIVE PRIORS FOR EXPRESSIVE AND TRANSFERABLE SKILLSS
While previous works chose IA on intuition Tirumala et al., 2019 2020 Galashov et al. 2019
Pertsch et al. |020: Wulfmeier et al.) 2019: Bagatella et al. 2022, Singh et al., 2020 Ajay et al.
2020; Liu al. 2022) we propose learning
Consider the information gating functions (IGFs)
introduced in Section 3 and depicted in Figure | Existing methods can be recovered by having the
IGF' perform hard attention: [xF( = m @ Xx. with m E {0, 1] [jdmm(x)) predefined and static, and O representing element-wi wize multiplication. In contrast, we propose performing soft attention
with m E [0, []dim(xx)) and learn based on: 1) the hierarchical KI KL-rregularized RI. objective
4

---

Published as a conference paper at ICLR 2023
(Equations and (2 ): 2) LIGF(m) -mm). H denoting entropy (calculated by turning m into
a probability distribution by performing Softmax over it), thereby encouraging low entropic, sparse IGFs (similar to Salter et al. (2099) applying related technique for sim2real transfer):
rk ( ar) anDKL ( (a) (T(a) 
0e
CA Mo. {m,
E Erap, koop(()
Σn t=
(3)
Σ am, H(m))
ET
With am, weighing the relative importance of the entropy and KL-regularized RL objectives for each attention mask {m: for each module using self-attention (e.g. TT"). denoting this set. Whilst soft attention does not eliminate dimensions in the same way that hard attention does, thus losing the strict connection with Theorems and 3.2 in practice it often leads to many O-attention elements (Salter et al. 2020: Mott et al., 2019). m; spans all history dimensions and supports intra-ohservation and intra-action attention. We train off- off-policy akin to SAC (Haarnoja et al. 2018b), sampling experiences from the replay buffer, approximating the return of the agent using Retrace (Munos et al. 2016) and double Q-learning (Hasselt 2010) to train our critic. Refer to Appendices [ anor full training details. Exposing IGFs to all available information Xk. we enable expressive skills that maximize the KL-regularized RL objective, with complex, potentially long-range, temporal dependencies (Theorem B.2). Encouraging low-entropic masks m, promotes minimal information conditioning (by limiting the IGF's channel capacity) whilst still capturing expressive behaviours. This is achieved by paying attention only where necessary to key environment aspects (Salter al. (022) that are crucial for decision making and hence heavily influence behaviour expressivity. Minimising the dependence on redundant information (aspects of the observation s, action a, or history x spaces that behaviours are independent to), we minimise covariate shift and improve the transferability of skills to downstream domains (Theorem 3.1 Consider learning the IGF of high-level prior To for a humanoid navigation task. Low-level skills π² could correspond to motor- aitor-primitives, whilst the high-level prior could represent navigation skills. For navigation, joint quaternions are not relevant, but the Cartesian position is. By learning to mask parts of the observations corresponding to joints, the agent becomes invariant and robust to covariate shifts across these dimensions (unseen joint configurations). We call our method APES, 'Attentive Priors for Expressive and Transferable Skills'.
4.1 TRAINING REGIME AND THE INFORMATION ASYMMETRY SETUP
We are concerned with investigating the roles of priors, hierarchy and IA for transfer in sequential task learning, where skills learnt over past tasks Psource (KC) are leveraged for transfer tasks Ptrans (K). While one could investigate IA between hierarchical levels (T m² as well as between policy and prior (x, No), we concern ourselves solely with the latter. Specifically, to keep our comparisons with existing literature fair, we condition 7 enabling expressive multi-modal behaviours to be discovered with respect to Sy (Tirumala et al., 2019) 2020; Wulfmeier et al., 2019). In this paper, we focus the role of IA between high-level policy x² and prior 7n for supporting expressive and transferable high-level skills between tasks. As is common, we assume the source tasks solved before tackling the transfer tasks. Therefore, for analysis purposes it does not matter whether we learn skills from source domain demonstrations provided by hardcoded expert an optimal RL agent. For simplicity, priors using variational behavioural cloning from expert policy samples:
and 2z, and share it with the prior, n² Th thus
discover skills and skill
(T, uu, {m,) )(ET)=
Σ )E(0,e)
LAPES (T, Ty. {m,) ( with Tą = 0, = 1, ~ (T)
(4)
Equation (4) can be viewed hierarchical KL-regularized RL in the absence of rewards and with two priors; the one we learn mo; the other the expert π.. See Appendix A.2 for a deeper discussion on the similarities with KL-regularized RL. We then transfer the skills and solve the transfer domains using hicrarchical KL-regularized RL (as per Equation 33). To compare the influence of distinct IAs for transfer controlled way, we propose the following regime: Stage 1) train single hierarchical policy æ in the multi-task setup using Equation 4 but prevent gradient flow from prior xo to policy. Simultaneously, based on the ablation, train distinct priors (with differing LAs) on Equation (4) to imitate the policy. As such, we compare various IAs influence on skill distillation and transfer in a controlled manner, as they all distil behaviours from the same policy; Stage 2) freeze the shared modules (x ' = and train newly instantiated " on the transfer task. By freezing m/, we assume the low-level skills from the source domains suffice for the transfer domains, often called the modularity assumption (Khetarpal et al. 2020a Salter et al. 2022). While appearing restrictive,
5

---

Published as a conference paper at ICLR 2023
Skill-level exploration for Sparse 2 corr. 4 rollouts, episodes unrolled horizontally. Corridors are colour coded and depth within them is denoted by shade; the darker the deeper. Hallway is white. Only the full setup leads to corridor-leve exploration.
Figure 3:
Hicudy
the increasingly diverse the source domains are (commonly desired in settings like lifelong learning (Khetarpal et al. (2020a) and offline RL), the increasingly probable the optimal transfer policy can be obtained by recomposing the learnt skills. If this assumption does not hold, one could either fine-tune x² during transfer, which would require tackling the catastrophic forgetting of skills (Kirkpatrick) et al.] (017). or train additional skills (by expanding dimensionality for discrete z spaces). We leave this as future work. We also leave extending APES to sub-optimal demonstration learning as future work, potentially using advantage-weighee regression (Peng et al. 2019), rather than behavioral cloning, to learn skills. For further details refer to Appendices A and
5 EXPERIMENTS AND RESULTS
Our experiments are designed to answer the following sequential task questions: (1) Can we benefit from both hierarchy and priors for effective transfer? (2) How important is TA choice between high-level policy and prior. Does it lead to an impactful practice, how detrimental is covariate shift for transfer?
trade-off? In (3) How favourably does APES automate the choice of IA for effective transfer? Which IAS are discovered? (4) How important hierarchy for transferring expressive skills? Is hierarchy necessary? We compare against competitive skill transfer baselines, primarily Tirumala et al. (2019 2020); |ertsch et al. (2020); Wulfmeier et al. (2019). on similar navigation and manipulation tasks for which they were originally designed and tested against.
5.1 ENVIRONMENTS
We evaluate on two domains: one designed for controlled investigation of core agent capabilities and the another, more practical, robotics domain (see Figure 2). Both exhibit modular behaviours whose discovery could yield transfer benefits. See Appendix C for full environmental setup details.
Marre The agent must traverse corridors in given ordering. We collect 4+ 10 trajectories from a scripted policy traversing any random ordering of two corridors (psouree (K)).. For transfer (Ptrans (AC)), inter- or extrapolated ordering must be traversed (number of sequential corridors {2, 4}) allowing us to inspect the generalization ability of distinct priors to increasing levels of covariate shift. We also investigate the influence of covariate shift on effective transfer across reward sparsity levels: s-sparse (short for semi-sparse), rewarding per half- colf-corridor completion; sparse, rewarding on task completion. Our transfer tasks are sparse corr and s-sparse corr.
Stack. The agent must stack a subset of four blocks over target pad in given ordering. The blocks have distinct masses and only lighter blocks should be placed on heavier ones. Therefore, discovering temporal behaviours corresponding to sequential block stacking according to mass, is beneficial. We collect 17.5 * 10 trajectories from scripted policy, stacking any two blocks given this requirement (psource (K)). The extrapolated transfer task (Ptrans (K)), called 4 blocks, requires all blocks be stacked according to mass. Rewards are given per individual block stacked.
5.2 HIERARCHY AND PRIORS FOR KNOWLEDGE TRANSFER
Table 2: Average return across 100 episodes (mean + standard deviation for 7 random seeds). Experiments that do not ((Hier-)RecSA) / do leverage prior experience, were ran for 10 vs 1.5 10 environment steps. APES employs hierarchy, priors and learns m for Th APES-(H2) S} do not learn m. The remainder do not employ priors.
Appreach
Paper
input
leara m
sparse coer (lat)
s-iparse 4 coe (ext)
blocks (ear)
APES
Xi 2014
0.92 : 0.04
7.03
0.06
3. 3. 1: 0
APES
Tirumala et al.
Xt-201 2012
0.15 +
3.79
1.10
APES H10
Xi-
X
0,25 1: 0.07
4.12 2 0.41
1:25 1 0.19
APES-HI
Ragotella et al
2022
X
0.80 + 0.02
6.33 = 0.13
3.13 # 0.09
APES-S
1:
2.96 0.32
2.05
Penkil
21A0
APES-mu.prin Hie- RecSAC
Trumala
2019
0.00 4: 0.00
2.30 0.11
0.00 0.00
0.07
0.01
RecSAC
Harnaja di mt.
20135
0.00 +: 0.00
0.08 0.01
0.01 2 0.00
Export

---

Published as a conference paper at ICLR 2023
Our full setup, APES, leverages hierarchy and priors for skill transfer. The high-level prior is given access to the history (as is common for POMDPs) and learns sparse self-attention m. To investigate
the importance of priors, compare against APES-no prior, baseline from Tirumala et al. (2019). with the full APES setup except without a learnt prior. Comparing transfer results in Table 2 we APES drastic gains highlighting the importance of temporal high-level behavioural priors. To inspect the transfer importance of the traditional hierarchical setup (with T (S,, Ze)), we compare peeioo
APES-no prior against two baselines trained solely on the transfer task. RecSAC represents history- dependent SAC (Haarnoja et al.] 2018d) and Hier-RecSAC' hierarchical equivalent from Wulfmeier et al. (2019). APES-no L.PES-no-prior prior has marginal benefits showing the importance of both hierarchy and priors for transfer. See Table for a detailed explanation of all baseline and ablation setups.
5.3 INFORMATION ASYMMETRY FOR KNOWLEDGE TRANSFER
Table 3: H(m) vs DKI. (Ta.(TT (Tr |
To investigate the importance of IA for transfer, we ablate over high-level priors with increasing levels of asymmetry (each in- put subset of the previous): APES-([120, HI0, H1, 53. S
Metic
H(/m)
Dez(a"lisdd
Dontan
ConnSack
ComyStack
APES-S
0.20//0.0
0.61
denoting an observation-dependeet high-level prior, Hi a history- dependent one, Xt- Crucially, these ablations do not learn m. unlike APES, our full method. Ablating history lengths is a nat- ural dimension for POMDPs where discovering belief states by
APES-HI
0.76/0.95
0.22/00.5
78/1/19
0.12//0.00 20
APUS-1120
2:08/2:20
0:11/00.7
0.26/1.20
0.16,/0.59
2.08/2.26
0.84/1.71
0.00/0.00
0.00/0.00
history conditioning is crucial (Thrun) 1999). APES-HI, APES-S
are hierarchical extensions of Bagatella et al. (202)); Galashov et al. 2019 respectively, and APES- H20 (representing entire history conditioning) is from Tirumala et al. (2020). APES-S is also an extension of Pertsch et al. (2020) with x (s 2:) rather than π (zz). Table shows the heavy IA influence, with the trend that conditioning on too little or much information limits performance. The level of influence depends on reward sparsity level: the sparser, the heavier influence, due to rewards guiding exploration less. Regardless of the transfer domain being interpolated or extrapolated, IA is influential, suggesting that IA is important over varying levels of sparsity and extrapolation.
5.4 THE EXXRESIIIYY-RRASFERBBBII
TRADE- OFF
To investigate whether Theo- rems 3.I and 3.2 are the rea-
Sparts
son for the apparent expressivity- mansferability trade-off seen in Table we plot Figure 4show- ing, on the vertical axis, the dis- tillation loss DKL (T || m6¹) at the end of training over Peource (KC), verses, on the hor- izontal axis, the increase in trans- P (KC)
14
Ixteacela Srarchy Perterrden Jiu Acc Tex etteggatoooe Sult
Trade- Off. Distillation loss transfer performance increase when pre-training H The level of information conditioning is shown by marker size. Absolute transfer performance is shown by marker colour (red - high; blue low). We fit a second order curve (in red) to show the general trend, per Theorems 3.1 and 3.2 that distilling more, by reducing IA, can hurt transfer, due to increased covariate shift. See Table for detailed tabular results.
Figure 4: Sppressivity-Trnsffraaabaa
fer performance (on Ptrans when initialising nh as a task
agnostic high-level policy pre- trained over Psource (IC) (instead
of randomly, is default). We ran these additional pre-trained n experiments to investigate whether covariate shift is the culprit for the degradation in transfer performance when conditioning the high- level prior of on additional information. By pre-training and transferring 'T we reduce initial trajectory shift and thus initial covariate shift between source and transfer domains (see Section 2.2). This is as no networks are reinitialized during transfer, which would usually lead to an initial shift in policy behaviour across domains. As per Theorem 3.1, we would expect a larger reduction in covariate shift for the priors that condition on more information. If covariate shift were the culprit for reduced transfer performance, we would expect larger performance gain for those priors conditioned demonstrates that the case in general, regardless of whether the on more more information. Figure 3
transfer domain is inter- or extra-polated. The trend is significantly less apparent for the semi-sparse domain, as here denser rewards guide learning significantly, alleviating covariate shift issues. We show results for APES- APES-(1120, 1110, 111, S} as each input is subset of the previous. These relations govern Theorems 3.1 and 3.2 Figure and Table B show that conditioning on more information improves prior expressivity, reducing distillation losses, per Theorem 3.2 These results together with Table show the impactful expressivity-rrasferabbii
trade-off of skills, controlled by IA (as per Theorems 3.1 and 3.2). where conditioning on too little or much information limits performance.
7

---

Published as a conference paper at ICLR 2023
5.5 APES: ATTENTIVE PRIORS FOR EXPRESSIVE AND TRANSFERABLE SKILLS
Zarideiilzee Alwitio
Store Aneetion
As seen in Table 2 APES, our full method, strongly outper- forms (almost) every baseline and ablation on each transfer do- main. Comparing APES with APES-H2O, the most compara- ble approach with the prior fed the same input (x;-20:t), we ob- serve drastic performance gains. These results demonstrate the im- portance of reducing covariate shift (by minimising information
Figure 5: APES attention for A0 plotted as 1og (m), for each domain (key on right; red and blue as high and low val- ues). We aggregate attention to the observation/action levels, e.g. logio (m,.) = log (2 ( m (with m,, the aggregate attention for 8 and m attention for the ith dimension of st). For intra-observatiinnaation attention visualisation, reporting val- ues per individual dimension, see Figure APES learns sparse, domain dependent, attention, primarily focusing on recent actions. conditioning), whilst still supporting expressive behaviours (by exposing the prior to maximal infor- mation), only achieved by APES. Table Shows H(m), each z/d mask's entropy (a proxy for the amount of information conditioning). vs DRi. (T |T0" (distillation loss), reporting max/min scores across all experiments cycles. We ran 4 random seeds but omit standard deviations as they were negligible. APES not only attends to minimal information (H(m)). but for that given level achieves a far lower distillation loss than comparable methods. This demonstrates APES pays attention only where necessary. We inspect APES" attention masks m in Figure (aggregated at the observation and action levels). Firstly, many attention values tend to 0 (seen clearly in Figure 7 aligning APES closely to Theorems 3.1 and 3.2 Secondly, APES primarily pays attention to the recent history of actions. This is interesting, as is inline with recent concurrent work (Bagatella et al. 2022) demonstrating the effectiveness of state-free priors, conditioned on a history of actions, for effective generalization. Unlike [Bagatelaa et al. (2022) that need exhaustive history lengths sweeps for effective transfer, our approach learns the length in an automated domain dependent manner. As such, our learnt history lengths are distinet for CorridorMaze and Stack. For CorridorMaze, some attention is paid to the most recent observation St, which is unsurprising as this information is required to infer how to optimally act whilst the end of corridor. In Figure 7 we plot intra-observation and action attention, and note that for Stack, APES ignores various dimensions of the observation-space spacc,, further reducing covariate shift. Refer to Appendix E.1 for an in-depth analysis of APES full attention maps.
453 Seath
.
14507 Septh
(ii=
5.6 HIERARCHY FOR EXPRESSIVITY
Table 4: Hierarchy Ablation
To further investigate whether hierarchy is necessary for effec- tive transfer, we compare APES-HI with APES-HI-flat, which has the same setup except with a flat, non-hierarchical prior
Transfer task
sparse 2
s-pune cor
APES
1
APES-HI -L-a
0.75 1: 0.07
5.65 1: 0.13
0.06 0.03
4.44
(conditioned on 1;) With a flat prior, KL-regularization must occur over the raw, rather than latent, action space. There-
Expert
fore, to adequately compare, we additionally investigate a hierarchical setup where regularization occurs only over the action-space. APES-HI-KL aL.a. Transfer results for CorridorMaze are shown in Ta seeds). Comparing APES-HI-KS
and -PES-HI-KII--l we see the benefits of hierarchi- cal prior, more significant for sparser domains. Upon inspection (see Section 5.7). APES-H KKL-flat is unable to solve the task as it cannot capture multi-modal behaviours (at corridor intersections). Con- trasting APES-II with APES-HI-KLL with alternate methods for multi modalityo,altt,, hierarchy may not be necessary.
we minimal benefits for latent regularization, suggesting
5.7 SKILL- EXPLORATION ANALYSIS
To gain a further understanding on the effects of hierarchy and priors, we visualise policy rollouts early on during transfer (5x 10x steps). For CorridorMaze (Figure with hierarchy and priors APES explores randomly at the corridor level. Hierarchy alone, unable to express preference over high-level skills, leads to temporally uncorrelated behaviours, unable to explore at the corridor level. The flat prior, unable to represent multi-modal behaviours, leads to suboptimal exploration at the intersection of corridors, with the agent often remaining static. Without priors nor hierarchy, exploration is further hindered, rarely traversing corridor depths. For Stack (Figure 6) APES explores at the block stacking level, alternating block orderings but primarily stacking lighter upon heavier. Hierarchy alone is unable to stack blocks with temporally uncorrelated skills, exploring at the intra-block stacking level, switching between blocks before successfully stacking, or often interacting with, any individual one.
8

---

Published as a conference paper at ICLR 2023
Figure 6: Skill-level exploration for 4 Blocks. End-effector and block rollouts depicted by dot- ted and dashed lines respectively, colour-coded per rollout. The end positions for gripper and blocks are represented by cross or cubes, re- spectively. Cube numbers represent their mass order with the heaviest. APES explores at the stacking level whilst hierarchy alone is unable to stack. See Figure 10 for video rollouts.
1) Prior &e. Hlenzryy
B) Elerarchy
6 RELATED WORK
Hierarchical frameworks have long history (Sutton et al.,
1999).
The options semi- MDP literature
explore hierarchy and temporal abstractions (Nachum et al. [2088]
Wulfmeier et al., 2020; Igl et al.
2019 (aatter et al.) [022 Kamat & Precup 2020; Harb al., 2018: Riemer al. 2018). Approaches
like Wulfmeier et al. (2019] 2020) use hierarchy to enforce knowledge transfer through shared
modules. Gehring et al. (2021) 1) use hierarchy to discover skills of varying expressivity levels. For
lifelong learning (Parisi et al. 2019, Khetarpal et al. (020b)) where number of skills increase over
time, it unelear how well these approaches will fare, without priors narrow skill exploration.
Priors have been used in various fields. In the context of offline-RL, Siegel et al. 2020); Wu et al. (2019) primarily use priors to tackle value overestimation (Levine et al., 2020). In the variational literature, priors have been used to guide latent-space learning (Hausman et al. [018 Ig et al. 2019)
Pertsch et al., 2020; Merel et al., 2018). Hausman et al. (2018) learn episodic skills, limiting their ability to transfer. [g] et al. 2019); Khetarpal et al. (2020a) learn options together with priors or interest functions, respectively. The primer is on on-policy, limiting applicability and sample efficiency. Both predefine information conditioning of shared modules, limiting transferability. Skills and priors have been used in model-based RL to improve planning (Xie et al. 2020; (hi et al. (022). Sikchi et al. (2022) use priors to reduce covariate shifts during planning. In contrast, APES ensures the priors themselves experience reduced shifts. In the multi-task literature, priors have been used to guide exploration (Pertsch et al. 2020: Galashov al. 2019: Siegel et al. 2020 Pertsch et al. 2022 Teh et al. 2017). yet without hierarchy expressivity in learnt behaviours is limited. In the sequential transfer literature, priors have also been used to bias exploration (Pertsch et al. 2020 AAjay et al.
2020: Singh et al. 2020] [Bagatella et al. [022] Goyal et al., 2019; Rao et al. 202 L et al. 2022]
yet either do not leverage hierarchy (Pertsch et al.] (020)) or condition on minimal information (Ajay et al. 2020 |Singh et al., 2020: Rao et al. 2021). limiting expressivity. Unlike APES, (Singh et al. (2020); |Bagatella et al. (2022) leverage blow- transformations to achieve multi-modality. Unlike many previous works, we consider the POMDP setting, arguably more suited for robotics, and learn the information conditioning of priors on based on our trrressivity-runsffrabittt theorems.
Whilst most previous works rely on IA, choice is primarily motivated by intuition. For example, Igl et al. (2019); Wulfmeier et al. (2019 2020) only employ task or goal asymmetry and Tirumala et al. (2019) Merel et al. (2020); Galashov et al. (2019) use exteroceptive asymmetry. [aattr et al. (2020) investigate a way of learning asymmetry for sim2real domain adaptation, but condition m on observation and state. We consider exploring this direction future work. We provide a principled investigation on the role of IA for transfer, proposing a method for automating the choice.
7 CONCLUSION
We employ hierarchical KL-regularized RL efficiently transfer skills across sequential tasks, showing the effectiveness of combining hierarchy and priors. We theoretically and empirically show the crucial regularized RL. Our experiments validate the importance of this trade-off for both interpolated and extrapolated domains. Given this insight, we introduce APES, 'Attentive Priors for Expressive and Transferable Skills' automating the IA choice for the high-level prior, by learning it in data driven, domain dependent, manner. This is achieved by feeding the entire history to the prior, capturing expressive behaviours, whilst encouraging its attention mask to be low entropic, minimising covariate shift and improving transferability. Experiments over domains of varying sparsity levels demonstrate APES' consistent superior performance over existing methods, whilst by- by-passing arduous IA sweeps. Ablations demonstrate the importance of hierarchy for prior expressivity, by supporting multi-moda. behaviours. Future work will focus on additionally learning the IGFs between hierarchical levels.
trade-off, controlled by IA choice, of skills for hierarchical KL-
9

---

Published as a conference paper at ICLR 2023
REFERENCES
Abbas Abdolmalcki, Jost Tobias Springenberg. Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv: 06900,6 2018.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.136111 2020.
Marco Bagatella, Sammy Joe Christen, and Otmar Hilliges. Sfp: State-free priors for exploration in off-policy reinforcement learning. 2022.
Alexandre Galashov, Siddhant M Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojeiech M Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in kl-regularized rl. arXiv preprint arXiv: 1905.01000002400 2019.
Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier. Hierarchical skills for efficient exploration. Advances in Neural Information Processing Systems, 34:11553-11564 2021.
Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, and Sergey Levine. Infobot: Transfer and exploration via the information bottle- neck. arXiv preprint arXiv:
2019.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In International Conference Machine Learning (ICML), 2018a. 8a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning (ICML), 2018b.
Tuomas Haarnoja, Aurick Zhou, Sehoon Ha, Jie Tan, George Tucker, and Sergey Levine. Learning walk via deep reinforcement learning. arXiv preprint /8/2. 2018c.
Tuomas Haarnoja. Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/2.05905, 2018 8d.
Jean Harb, Pierre- Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not option: Learning options with a deliberation cost. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
H. V. Hasselt. Double Q-learning Advances in Neural Information Processing Systems (NIPS). pp. 2613-2621, 2010.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. International Conference on Learning Representations, 2018.
Maximilian Igl, Andrew Gambardella, Jinke He, Nantas Nardelli, N Siddharth, Wendelin Böhmer, and Shimon Whiteson. Multitask soft option learning. arXiv preprint arXiv:1904.01033, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzation with gumbel-softmax. arXiv preprint arXiv: 1611.0 144, 2016.
Anand Kamat and Doina Precup. Diversity-enriched option-critic. arXiv preprint arXiv:20/1.02565 2020.
Hilbert J Kappen, Vieenç Gómez, and Manfred Opper. Optimal control a graphical model inference problem. Machine learning, 1792):159-182 2012.
Khimya Khetarpal, Martin Klissarov, Maxime Chevalier-Boisee
Pierre-Lue Bacon, and Doina Precup. Options of interest: Temporal abstraction with interest functions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 4444 4451, 2020a.
10

---

Published as a conference paper at ICLR 2023
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. arXiv preprint arXiv:2012.13499, 2020b.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Bariskka-aawikkaa, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114
(13):3521-3526, 2017.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint arXiv:2005. 0/000000000003,,, 2020.
T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1rXiv::509.0022
2015.
Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, SM Ali Eslami, Daniel Hennes, Wojciech M Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, et al. From motor control to team play in simulated humanoid football. Science Robotics, 7669):eab00235, 2022.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid con- trol. arXiv preprint 1rX 2018.
Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. Catch carry: reusable neural controllers for vision-guided whole-body tasks. ACM Transactions Graphics (TOG), 39(4):39-1, 2020.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland. G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533. 2015.
Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. Advances in Neural Information Processing Systems, 32, 2019.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare G Bellemare. Safe and efficient off-policy reinforcement learning. arXiv preprint arXiv: 1600:::66600000 02647, 2016.
Ofir Nachum, Honglak Lee, Shixiang Gu, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. In Advances in Neural Information Processing Systems, volume 2018-Decem, pp. 2018. URL[https: http//tteessee google httpe////w///ee///eeeeeeeeeeeeee
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightee regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 2019.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010.11990 2020.
Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph Lim. Guided reinforcement learning with learned skills. 2021.
Matthias Plappert, Marcin Andrychowiez, Alex Ray. Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Macick Chociej, Peter Welinder, et al. Multi-goal reinforcement Iearn- ing: Challenging robotics environments and request for research. arXiv preprint 100,009400 2018.
Dushyant Rao, Fereshteh Sadeghi, Leonard Hasenclever, Markus Wulfmeier, Martina Zambelli, Giulia Vezzani, Dhruva Tirumala, Yusuf Aytar, Josh Merel, Nicolas Heess, et al. Learning transferable motor skills with hierarchical latent mixture policies. arXiv preprint 11X 2021.
K. Rawlik, M. Toussaint, and S. Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. Robotics: Science and Systems (RSS), 2012.
11

---

Published as a conference paper at ICLR 2023
Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. Advances in neural information processing systems, 31, 2018.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 627-635. JMLR JML. Workshop and Conference Proceedings,
2011.
Sasha Salter, Dushyant Rao, Markus Wulfmeier, Raia Hadsell, and Ingmar Posner. privileged reinforcement learning for domain transfer. 2019.
Attention
Sasha Salter, Dushyant Rao, Markus Wulfmeier, Raia Hadsell, and Ingmar Posner. Attention- privileged reinforcement learning. In Conference on Robot Learning, 2020.
Sasha Salter, Markus Wulfmeier, Dhruva Tirumala, Nicolas Heess, Martin Riedmiller, Raia Hadsell, and Dushyant Rao. Mo2: Model-based offline options. Conference on Lifelong Learning Agents, 2022.
J. Schulman, P. Abbeel, and X. Chen. Equivalence between policy gradients and soft Q-learning. arXiv preprint arXiv: 704.06440, 2017.
Lucy Xiaoyang Shi, Joseph J Lim, and Youngwoon Lee. Skill-based model-based reinforcement learning. arXiv preprint arXiv:2207.07566 2022.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396 2020.
Harshit Sikchi, Wenxuan Zhou, and David Held. Learning off- off-policy with online planning. In Conference on Robot Learning, pp. 1622-1633. PMLR, 2022.
David Silver, Julian Schrittwisse,, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge, nature, 550(7676):354-533 2017.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv:2011.10023 2020.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3 (1):9-44, 1988.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1 112(122)181-2111 1999.
Yee Whye Teh, Victor Bapst, Wojciech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. arXiv preprini 17Xiv:1707.0414 2017.
Sebastian Thrun. Monte carlo pomdps. Advances in neural information processing systems, 12, 1999.
Dhruva Tirumala, Hyeonwoo Noh, Alexandre Galashov, Leonard Hasenclever, Arun Ahuja, Greg Wayne, Razvan Pascanu, Yee Whye Teh, and Nicolas Heess. Exploiting hierarchy for learning and transfer in kl-regularized rl. arXiv preprint 19X300:903 2019.
Dhruva Tirumala, Alexandre Galashov, Hyeonwoo Noh, Leonard Hasenclever, Razvan Pascanu, Jonathan Schwarz, Guillaume Desjardins, Wojciech Marian Czarnecki, Arun Ahuja, Yee Whye Teh, et al. Behavior priors for efficient reinforcement learning. arXiv preprint MxXi:20100112244 2020.
E. Todorov. Linearly-solvable Markov decision problems. In Advances in Neural Information Processing Systems, pp. 1369-1376. MIT Press, 2007.
12

---

Published as a conference paper at ICLR 2023
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 5026- 5033. IEEE, 2012.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1XXXv:9911.13661 2019.
Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg. Michael Neunert, Tim Hertweck, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Compositional Transfer in Hierarchical Reinforcement Learning. (1), 2019. URL htthttttp/aate http::::/axxiv.or///aaaa
11906.1222
Markus Wulfmeier, Dushyant Rao, Roland Hafner, Thomas Lampe, Abbas Abdolmaleki, Tim
Hertweck, Michael Neunert, Dhruva Tirumala, Noah Siegel, Nicolas Heess, al. Data-efficient hindsight off-policy option learning, arXiv preprint arXiv:2007. 15588, 2020.
Kevin Xie, Homanga Wharadhwaj. Danijar Hafner, Animesh Garg, and Florian Shkurti. Latent skill planning for exploration and transfer. arXiv preprint uxXiv2201..38997 2020.
13

---

Published as a conference paper at ICLR 2023
A METHOD
A.1 TRAINING REGIME
In this section, algorithmically describe training setup. We relate each training phase the principle equations in the main paper, but note that Appendices and AA.3 A.33Juttinee outline a more detailed version of these equations that were actually used. We note that during BC, apply DAGGER (Ross) 201 as per Algorithm improving learning rates. For further details refer to Appendix
et al.
Algorithm APES training regime
Algorithm 2 collect
1: # Full training and transfer regime. For BC, gradients are prevented from flowing from No In practice To = {T)) (i (O....... multiple trained priors. During transfer. n" is reinitialized.
1:
# Collects experience from either or ne applying DAGGER at a given rate if instructed, and updates R₃- env accordingly.
2:
2: function COLLECT(TI. Rj. enw,
3: # Behavioral Cloning
dag=False, r=1)
4: Initialize: policy a, prior To. replay Roes DAGGER
env.observation() 4: Te &- enw.expert()
rate environment env
5: for Number BC training steps do
a; (x(x)
6: Rbbs env collect(1,, Rbcr env, True. r)
6:
a, + Tx)
7: n. Tu < BC..pdate(T, To, Rbc) Eq.
a Bernouli((aa, a 1 r()
8: end for
8:
x', Tky enc + env.step(a)
9: # Reinforcement Learning
if dag then
Initialize: high level policy =H crities QkE(1,23, re-
10:
aʃ + a.
play Ryi. transfer environment env; for Number RL training steps do Ryg, enUY + collect(T, Rr,, ene))
11:
else
12:
a a;
12:
13:
end if
nH Ro-pccy update( To. R,d) # Eq. Qx < <L=RL_critic_uup Sucraiicc.udate(Q,... R,,) # Eq.
14:
Rg R.,.udatt xx,,,
14:
15:
return Ry. EnV
15:
end for
16:
end function
A.2 VARIATIONAL BEHAVIORAL CLONING AND REINFORCEMENT LEARNING
In the following section, we omit APES" specific information gating function objective F for simplicity and generality. Nevertheless, it is trivial extend the following derivations to APES.
Behavioral Cloning (BC) and KL-Regularized RL, when considered from the crriational-inffreen perspective, share many similarities. These similarities become even apparent when dealing with hierarchical models. particularly unifying choice of objective functions for BC and RL that fit with off off-policy, generative, hierarchical RL: desirable for sample efficiency, are:
Lba(T, (m:)iet) =
(π(T) - DEL NEI
m((7),, Cal(T, (m,) (R)() (m:)
(5)
Loc, corresponds to the KL-divergence between trajectories from the policy, म, and various priors, T₁ For BC, i {0, e), denote the learnt, uniform, and expert priors. For BC, in practice, we train multiple priors in parallel: To = {T, Jiv (O. We leave this notation out for the remainder of this section for simplicity. When considering only the expert prior, this is the reverse KL-divergence, opposite to what is usually evaluated in the literature, (Pertsch et al. 2020). Lyt, refers to lower bound on the expected optimality of each prior log Pr, O= 1); O denoting the event of achieving maximum return (return referred to as R(-)); refer to (Abdolmaleki et al. (018)) appendix B.4.3 for proof, further explanation, and necessary conditions. During transter using RL, do not have access to the expert or its demonstrations (i C := i {0, u)).
For hierarchical policies, the KL terms are not easily evaluable. DEL (π(TT || (T(T))
Σ, (r) [DRL. (m (2 (xx) ) || n. (z, (x)) + (e.((x) [DLL (a Xx, 24) o 7f XE, (.))
14

---

Published as a conference paper at ICLR 2023
Tirumala al. 2019). is commonly chosen upper bound. If sharing modules, e.g. Th. = x², or using non- non-hierarchical networks, this bound can be simplified (removing the second or first terms respectively). To make both Eq. (5) amendable to off- off-policy training (experience from {Te: mb). for BC/RL respectively; тh representing behavioral policy), we introduce importance weighting (IW), removing off-policy bias at the expense of higher variance. Combining all the above with additional individual term weighting hyperparameters, {37, ( we attain:
D (T())())( = -Ey(r) Σ v911 (3) C..(((())))) + 3r (2.(())))) (Ci,(ay Xtx, 2
B (2.1x))) [T (at |xi, 2z: k:]]
[
TT)
cr
22-0
S II St Cnc(l)= log
n(a,)x,,
TTi.c (1)
- -DKL((((F)((T((())
Σ
(π(F))l<((T))
KL
(6)
iE (0,0ue.))
pp(K). mo(r)
= [lo 2 Br.(T)
Enma [t] The (x1,, ar)
Σ
D (T(F)]|(((Tt)
iE(0,u)
Where Do(r) KL. (T())](T;TT) (for {0], [8] 1) is an unbiased estimate for the aforementioned upper bound, using q's experience. T the IW for timestep i, between æ and arbitrary policy n. n²[4 the 9th element of v"; the cumulative IW product at timestep Equations 6 7 are the BC/RL lower bounds used for policy gradients. See Appendix for a derivation, and necessary conditions, of these bounds. For BC, this bounds the KL-divergence between hierarchical and expert policies, æ, Te- For RL, this bounds the expected optimality, for the learnt prior policy, To. Intuitively, maximising this particular bound, maximizes return for both policy and prior, whilst minimizing the disparity between them. Regularising against uninformative prior, Nu. encourages highly-entropic policies, further aiding at exploration and stabilising learning (Igl et al. 2019).
In RL, IWs are commonly ignored (Lillicrap al. 2015 Abdolmalcki et al. 2018 Haarnoja et al. 2018b), thereby considering each sample equally important. This is also convemient for BC, as IWs require the expert probability distribution: not usually provided. We did not observe benefits of using them and therefore ignore them too. We employ module sharing (Té = nt; unless stated otherwize), and freeze certain modules during distinct phases, and thus never employ more than 2 hyperparameters, B, at any given time, simplifying the hyperparameter optimisation. These weights balance an exploration/explootttoo trade-off. We use categorical latent space, explicitly marginalising over, rather than using sampling approximations (Jang et al. (016)). For BC, we train for epoch (referring to training in the expectation once over each sample in the replay buffer).
A.3 CRITIC LEARNING
The lower bound presented in Eq. 7 is non-differentiable due to rewards being sampled from the environment. Therefore, as is common in the RL litcrature (Mnih et al. 2015 Lillicrap et al. 20155). approximate the return of policy with critic, Q. To be sample efficient, we train in off off-policy manner with TD-Icarning (Sutton 1988) using the Retrace algorithm (Munos al. 2016) to provide low-variance, low-bias, policy evaluation operator:
QTT = Q' Q'(x,, ar, k)) +
(xi,ar,A
Σ
o
(x(x) + = (a)x)+1.)) a (1112,))
(e(xxt1,,a - Q ('(x), as k))
1=1
(8)
((Q) = Ep (T(7)
(Q(x), ac, k)) arg min arggin min(Qre))
cj - c
(9)
Q
#=1+1
For proof refer Appendix B.3
15

---

Published as a conference paper at ICLR 2023
Where Q!" represents the policy return evaluated via Retrace. Q is the target Q-network, commonly used to stabilize critic learning (Mnih et al., 2015), and is updated periodically with the current Q values. IWs not ignored here, and
clipped between [0, 1] to prevent exploding gradients, (Munos et al. 2016). To further reduce bias and overestimates of our target, O we apply the double Q-learning trick, (Hasselt, 2010), and concurrently learn two target Q Q-networks, Q Our critic is trained minimize the loss Eq. (9) which regularizes the critic against the minimum of the two targets produced by both target networks.
B THEORY AND DERIVATIONS
In this section we provide proof' for the theory introduced in the main paper and in Appendix A
B.I THEOREM 1.
Theorem 1. The more random variables network depends on, the larger the covariate shift (input distributional shift, here represented by KL-divergence) encountered across sequential tasks. That is, for distributions p.
DKL (p(b) g(b)) 2 DKL (p(c) g(c)) with b (bo, b1. ..., bn) and c C
(10)
Proof
Ey(b) log p(b)
DKL (p(b) ll {bb)
<(b)
log p(dlc) p(c)
g(dlc) g(c) ]
r(()
with d
(II)
p(c)
Baaa Tog p(d[c)
)]
Ep(c) Extale) [1] log
+ Ep(e)
q(c)
<(dcc)
DKL (p(c) || ((c)) + Ep(c) [DL (p(dlc) g(dlc))]
2 DKL (p(c) || ((c)) gäven Ey(e) [DLL (p(dlc) |l [(d[c)) 20
B.2 THEOREM 2.
Theorem 2. The more random variables a network depends on, the greater its ability to distil knowledge in the expectation (output distributional shift between network and target distribution, here represented by the expected KL-divergence), That is, for target distribution p and network with outputs a and possible inputs c, such that (bu, bi,..., bn. and C
Eg(e/d) [DLL (p(alb) || q((aa))))))) s DKL (p(alb) || q(ald)) with e e
(12)
Proof
DKL (p(alb) || ((ald)) Ep(a|b) log p(a[b)
<(a)d)
Ep(a/b) [log ((a[b) log Ey(o [g(alc)]] with e e
(13)
o((e)) log ((a[bb
given Jensen's 's Inequality
q(aic)
Ey(eja) [DeL (p(a[b) q(a[c))]
B.3 HIFRARCHICAL KI.-DIVERGENCE UPPER BOUND
All proofs in this section ignore multi-task setup for simplicity. Extending this scenario is trivial.
16

---

Published as a conference paper at ICLR 2023
Upper Bound
DKL (5(r) 11 (+)T)) - Σ (x) [DLL (+ (ze(xe) || ( (zelxx))
(14)
(x)xx)) [DKL (a, 2,) (a/ 2c)]]]
Proof
(T)
DKL (I(T) ([())) Bx(())
log
[T))
p(so) II, (lppp((Sttt (x: at) m(aa(x)))
Bx((r)
log
p(so) II. p( Xx, ar) m (aglxx))
(ar (xt)
Bn(r) log II
(ar
Σ (r)(r) [DeL (T(a)/x))
(a,xxx))
$ Σ (D)K) (a(((x)))
||
n. (a)))))) +
(15)
Ex(n/(x)) [DKL (z(z)) ad)
x (Z) X5, (z, ac)] T( Xx
T (Zr Xe; ac)) T; (Ze [x,, ax))
= Barr) (e
(: x.) log
+ log
En((7) (DrL. (T 2o (x))
Tie (ar: Zr [x)]]]
Σ (Dt (TH (2e(2c)
क. (ze xx))
Fr (xx())) [DRL. (ar 2:)
T/ (a.lxx,2)))
B.4 POLICY GRADIENT LOWER BOUNDS
B.4.1 IMPORTANCE WEIGHTS DERIVATION
Del (T())][=(()) = ub(DrI. (=(T) m,(r)))
(16)
For ,8- = 1, where ub(DRL (I(7) || m. (T))) corresponds to the hierarchical upper bound intro- duced in Appendix A.2
Proof
ub(DRI. (T(T) mT(T))) = Σ En(r [KL ( (2((x)))
T (z, (x:)
+ (n
[KL (a xz, zz) || -.- (a. xx.))]]]
= Σ
[DKL (It (zz(xx) || ا.f (z)(x))
(())
+ E, H (a.)))) DKL (a (ar Xt, 2z) || ( Xx. 2:))
(a
(17)
= Σ E (
[DL. (T (z.(x.) T." (ze (xx))
m(aia))) (())))))
+ E, (2, (x.)
[DLL (a xx, || प. (ar x:))
(g(r)) (TTT)][
17

---

Published as a conference paper at ICLR 2023
B.4.2
BEHAVIORAL CLONING UPPER BOUND
Σ
D
KL. ( (T(())(T((T))
iE (0.,e))
(18)
for 87 17 3, 2
Proof
DRL. (T(7) me(TT) S
Σ
DEL. (m(T) (T((T)
(0,
Σ
ub(DKr. (m(T) ll [+(T))
i (0,a,c)
(19)
Σ
Der (T(T)]]<((TT) for 87
(E(0,,,))
D KL (T(T))(<(())) for 8 2
<
tE(O,a,e)
The last line holds true as each weighted term in (i(T) which are positive.
(T(T)]] corresponds to KL-divergences
B.4.3
REINFORCEMENT LEARNING UPPER BOUND
E P(() [log(O 1|T, k)] > Bro(T) Σ 1 (x) mD(r)
Σ
x (x(T)l(((TT)
(20)
(E(0.u)
for 37, 21 and rk <
Proof
Ep(c) [log(O 1|T 2 Bas(r) Ro(F)
Σ (ns T. (x, at)
DKL. (x((T(()) || [:([T)
2 Br)(T)
[x (x ag)
( (T())((()]) for 8 8 =
2 Br((T)
1 [t] TT (Xt, ac)
( (T())()(())) for 87,39 2
2 Exs(T)
Σ [mb Fh (xt, an)
Σ
D D (T(T))(T(7))
WE (0.u)
for 3 21
(21)
For line proof see (Abdolmaleki et al. 2018). The final 2 lines hold due to positive KL-divergences..
C ENVIRONMENTS
Here we cover each environment setup in detail, including the expert setup used for data collection.
C.1 CORRIDORMAZE
Intuitively, the agent starts at the intersection of corridors, at the origin, and must traverse corridors, aligned with each dimension of the observation space, in given ordering. This requires the agent to reach the end the corridor (which we call calf-corridor cycle), and return back to the origin, before the corridor is considered complete.
18

---

Published as a conference paper at ICLR 2023
E {0,1), p(so) = 0" k one-hot task encoding, [0, 1], spmrsssparse "kk (x,, a,)
if
agent has correctly completed the entire or half-corridor cycle else 0, guuso (x,,ar) ar)
1 if task complete else Task considered complete when a desired ordering of corri- dors have been traversed. = represents the number of corridors in experiments. / = 6, the lengths of each corridor. Observations transition according to deterministic transition function s/41 = f( az). Je corresponds to the index of the current corridor that the agent is in (i.e. j. = if the agent is in corridor 0 at timestep t). s' corresponds to the 1th dimension of the observation. Observations transition incrementally or decrementally down a corridor, and given observation dimen- sion s², if actions fall into corresponding transition action bins Vine, ddec. We define the transition function as follows:
s + 1, if ti (a (a.).)).
f(( a²)
S 1,
elif &dec (a.: otherwise.
(22)
0,
2/ (a. = aa in c, (j + 0.5 (.55ww//c) wc)c)), e (a, = bool(a, in Lj/c. (2. j- 1-:j--0.. 0.5 wc//c) The smaller the w parameter, the narrower the distribution of actions that lead to transitions. As such, w, together with 9ą controls the exploration difficulty of task k. We set w = 0.9. We constrain the observation transitions not transition outside of the corridor boundaries. Furthermore, if the agent is at the origin, s 0 (at the intersection of corridors), then the transition function is ran for all values of jt. thereby allowing the agent to transition into any corridor.
C.1.1 EXPERT SETUP
The expert samples actions uniformly within the optimal action bin range, from Eq. (22). that leads to the optimal state transition, traversing the correct corridor in the correct direction, according the task, which corridors have been traversed, and which remain.
C.2 STACK
This domain is adapted from the well known gym robotics FetchPickAnddPlaee environment (Plappert et al., 2018). The following modifications were made: 1) 3 additional blocks were in- troduced, with different colours, and goal pad, 2) object spawn locations were not randomized and were instantiated equidistantly around the goal pad, see Fig. 2 3) the number of substeps was increased from 20 to 60, this reduced episodic lengths, 4) transparent hollow rectangular tube was placed around the goal pad, to simplify the stacking task and prevent stacked objects from collapsing due to structural instabilities, 5) the was always spawned over the goal pad, see figure Fig. 2 6) the observation space corresponded to gripper position and grasp state, as well as the object positions and relative positions with respect to the arm: velocities were omitted as access to such information may not be realistic for real robotic systems. k = one-hot task encoding,
sparse (X,, az) = if correct object has been placed on stack in correct ordering else rk
C.2.1 EXPERT SETUP
The expert is set up to stack the blocks in the given ordering. Each individual block stacking cycle consists of six segments: 1) Move the gripper position a target location 20cm directly above the block, keeping the gripper open; 2) Vertically lower the gripper to 5mm over the block, keeping the gripper open; 3) Close the gripper until the object is grasped; 4) Vertically raise the gripper to 20cm above the initial block position, keeping the gripper closed: 5) Move the gripper to target location 20cm above the target pad, with the gripper closed; 6) Open the gripper until the object is dropped onto the target pad.
We use Mujoco's 's Todorov et al., 2012 PD controller, given target relative desired gripper position, which coupled with Mujoco's inverse kinematics model, produces desired actions. We apply gains of 21 to these actions. One the target location is reached for given stage, we proceed to the next. When opening or closing the gripper we apply actions of 0.05, -0.1, for stages 3 and 6. For stage 3, we continue closing the gripper until there are contact forces between the gripper and cube. For stage 6, continue opening the gripper until the block has dropped such that it is within 14em of the target pad. To prevent fully deterministic samples, which can be problematic for behavioural cloning, we inject noise into the expert actions. Specifically, we add Gaussian noise with a diagonal covariance
19

---

Published as a conference paper at ICLR 2023
Table 5: Feedforward Module, "(0)
(H.L)
hidden layers
(512, 512)
hidden layer activation
relu
output activation
linear
Table 6: Full experimental setup. Describes inputs to each module, level over which KL-regularization occurs, which modules are shared, and which are reused across training and transfer tasks.
Name
learn m
H 5D
7
KL level
no
reused modules
APES
20:
X1
ao no ao To π
n
-PES-H20
X;
20:1
81
S:
Xk
2
AIO
X
10:1
X
no no
APES-HI
X
81
S:
X
mp To 7
APES-S
St
X4
mi To m
APES-nO.prion
XF
Heer-ReeSAC
8:
X
ResSAC
APES- HI-KL-a
Ni-
St
8:
XA
X
40 TT
APES-I1I-fat
St
X
and standard deviation of 0.2 per dimension. We do not apply noise to the gripper closing or opening actions.
D EXPERIMENTAL SETUP
We provide the reader with the experimental setup for all training regimes and environments below. build off the softlearning code base (Haarnoja et al., 2018b). Algorithmic details not mentioned in the following sections are omitted as are kept constant with the original code base. For all experiments, we sample batch size number entire episodes of experience during training.
D.1 MODEL ARCHITECTURES
We continue by outlining the shared model architectures across domains and experiments. Each policy network (e.g. "" x/, !! m6) comprized of feedforward module outlined in Table 5
The softlearning repository that we build off (Haarnoja et al., (0188)) applies tanh activation over network outputs, where appropriate, to match the predefined outpute ranges of any given module. The critic is also comprized of the same feedforward module, but is not hierarchical. To handle historical inputs, we tile the inputs and flatten, to become one large input 1-dimensional array. We ensure the input always remains of fixed size by appropriately left padding zeros. For n x we use a categorical latent space of size 10. We found this dimensionality sufficed for expressing the diverse behaviours exhibited in domains. Table describes the setup for all the experiments in the main paper, including inputs to each module, level over which KL-regularization occurs (2 or a), which modules shared (e.g. n and n²f), and which modules reused across sequential tasks. For the covariate-shift designed experiments in Table g we additionally reuse n (or T for RecSAC) across domains, and whose input x,. For all the above experiments, any reused modules are not given access to mask-dependent information, namely task-id (k) and exteroceptive information (cube locations for Stack domain). This choice ensures reused modules generalize across task instances.
=
D.2 BEHAVIOURAL CLONING
For the BC setup, we use a deterministic, noisy, expert controller to create experience to learn off. We apply DAGGER (Ross et al. 2011) during data collection and training of policy π as found this aided at achieving high success rate at the BC tasks. Our DAGGER setup intermittently during data collection, with a predefined rate, samples an action from instead of ne, but still saves BC
20

---

Published as a conference paper at ICLR 2023
target action d, as the one that would have been taken by the expert for x₃. This setup helps mitigate covariate shift during training, between policy and expert. Noise levels were chosen to be small enough so that the expert still succeeded at the task. We trained our policies for one epoch (once over each collected data sample in the expectation). It may be possible to be more sample efficient, by increasing the ratio of gradient steps data collection, but did not explore this direction. The interplay we use between data collection and training over the collected experience, is akin to the RL paradigm. We build off the softlearning code base (Haarnoja al. 2018b), please refer to it for details regarding this interplay.
Table 7: Full Training Setup
(a) Behavioural Cloning Setup
(b) Reinforcement Learning Sctup
Environment
CorridorMaze
Stack
Environment
CorridorMaze
Stack
(() learning rate
3e
3e
Reward Type
sparse
semi-sparse
sparse
* categorical size
10
10
Transfer task
2 corridor
4 corridor
4 blocks
8 history-depth
24
le
1e
Q learning rate
3e -6
3e 5
3c" 5
(9g
le-2
le
learning
3e
3e
3c
18
8/
le
le
5e
/8 = /6
/85/"
1e
0
5e
O'm
le-1
1e
Q update rate
6e
6e
DAGGER rate
0.1
0.1
Retrace X
0.99
0.99
0.99
batch size
128
128
hatch size
128
128
128
episodic length
24
35 episodic length
30
60
63
Refer to Table 7a for BC algorithmic details. It is important to note here that, although we report five 9 hyper-parameter values, there are only two degrees of freedom. As we stop gradients flowing from To to æ, choice of So is unimportant (as long as it is not 0) as it does not influence the interplay between gradients from individual loss terms. We set these values 1. 8 absolute value is also unimportant, and only its relative value compared to 3 and 8 matters. We also set & to 1. For the remaining two hyper-parameters, 8ü, 8%, we performed hyper-parameter sweep over three orders of magnitude, three values across each dimension, to obtain the reported optimal values. In practice To = (ie (N)....N) multiple trained priors each sharing the same Bo hyper-parameterss parameters. For am, denoting the hyper-paramter in Equation (3) weighing the relative contribution of the n6' 's self-attention entropy objective IxF((x) with relation to the remainder of the RL/BC objectives, we performed a sweep over three ordered of magnitude (1c", 1e 1E 1.) This sweep was ran independent of all the other sweeps, using the optimal setup for all other hyper-parameter the hyper-paramete with lowest 1x (x ( ), H(m) combination. Four seeds were ran, as for all experiments. We observed very small variation in learning across the seeds, and used the best performing seed to bootstrap off for transfer. We separately also performed hyper-parre parameter sweep over ı learning rate, in the same way as before. We did not perform a sweep for batch size. We found for both BC and RL setups, that conditioning on entire history for π H was not always necessary, and sometimes hurt performance. We state the history lengths used for ' M for BC in Table 7a This value was also used for both n and Q for the RL setup.
We chose
We prevent gradient flow from F. to n. to ensure as fair a comparison between ablations as possible: each prior distils knowledge from the same, high performing, policy æ and dataset. If we simulta- neously trained multiple и and Ta pairs (for each distinct prior), is possible that different learnt priors would influence the quality of each policy which knowledge is distilled off. In this paper, we are not interested in investigating how priors affect π during BC, but instead how priors influence what knowledge can be distilled and transferred. We observed prior KL-distillation loss convergence across tasks and seeds, ensuring a fair comparison.
D.3 REINFORCEMENT LEARNING
During this stage of training freeze the prior and low-level policy (if applicable, depending the ablation). In general, any reused modules across sequential tasks are frozen (apart from or for the covariate shift experiments in Table 8) Any modules that not shared (such as / for most
21

---

Published as a conference paper at ICLR 2023
experiments), are initialized randomly across tasks. The RL setup is akin the softlearning repository (Haarnoja et al. 2018b) that we build off. We note any changes in Table 7b We regularize against the latent action level for, depending share low level policies, or use pre-trained modules (low-level policy and prior). Therefore, we only ever regularize against, at most, two hyper-parameten the same way as previously. We did not sweep over Retrace X, batch size, or episodic length. For Retrace, we clip the importance weights between [0, 1], like in Munos al. (2016), and perform returns rather than n-step. We found the Retrace operator important for sample-efficien learning.
the ablation, whether not our models are hierarchical,
Hlyper-parameter sweeps are performed in
E FINAL POLICY ROLLOUTS AND ANALYSIS
F. ATTENTION ANALYSIS
We plot the full attention maps for APES in Figure 7 including intra- observation and action at- tention. For CorridorMaze, at-
CorriorMaze Attertion
Stack Attention
tention is primarily paid to the most recent action a, For most observations in the environment (excluding end of the corridor or corridor intersection observa- tions), conditioning on the pre- vious action suffices to infer the optimal next action (e.g. continue that APES has learnt such an attention mechanism. For the remainder of the environment observations (such as corridor ends), conditioning on (a history of) observations is necessary to infer optimal behaviour. As such APES pays some attention to observations. For Stack, attention is primarily paid to a short recent history of actions, with the weights decaying further into the past. Interestingly, attention over actions corresponding to opening/closing the gripper (the bottom row of Figure decay a lot quicker, suggesting that this information is redundant. This makes sense, as there exists strong correlation between gripper actions at successive time-steps, but this correlation decays very quickly. Additionally, APES does not pay attention to observations corresponding to gripper position (the final 3 observation rows in Figure 7 this can be inferred from the remainder of the observation-space as well as the recent history of gripper actions.
History Dopth
History Death
H plotted as (m), for each Figure 7: APES attention for no
domain (key on right; red and blue as high and low values). APES learns sparse, domain dependent, attention.
traversing the depths of corridor). Therefore, it is understandable
E.2 COVARIATE SHIFT ANALYSIS
Table 8: Covariate Shift Analysis. In general, reduced IA benefits more from reduced covariate shift. Sparse domains suffer more from shift, seen by clearer IA co- variate trends.
In Table 8 we report additional return (over randomly initialised " achieved by pre-training (over Psource (K)) and transferring a task-agnostic high-level pol- icy n during transfer (Ptyone (K)). For experiments that are not hierarchical we pre-train equivalent non-hierarchical agent. Theorem 3.1 suggests we would expect a larger improvement in transfer performance for priors that condition on more information. Table confirms this trend demonstrating the importance of prior covariate shift in transferability of behaviours. This trend is less apparent for the semi-sparse domain. Additionally, for the interpolated transfer task (2 corridor), the solution is entirely in the support of the training set of tasks. Naively, one would expect pre- training to fully recover lost performance and match the most performant method. However, this is not the case as the critic, trained solely on the transfer task, quickly encourages sub-optimal distribbbtt behaviours.
CorriderMaze
Stack
interpolate
extrapolate
extrapolate
sparse
scmi-sparse
sparse
Approach
2 corridor
corridor
blocks
APES-H20 APES-H11
0.28 + 0.03
0.12 + 0.22
0.84 + 0.06
0.24 1: 0.07
0.62 0.30
1.20 : 0.31
APES
0.13 0.02
0.34 1 0.32
0.22 1 0.24
APES-S
0.00 + 0.00
0.48 + 0.21
0.00 + u.17
APES no prior
0.00 1.011:00 0.00
0.09 1: 0.08
0.49 1. 0.26
Hier-RecSAC RecSAC
0.00 0:0-::0 0.00
0.02 + 0.03
0.00 0.01
0.00 0.00
0.27
0.13
0.19 0.05
Expert
0
0
22

---

Published as a conference paper at ICLR 2023
Figure 8: CorridorMaze 4 corridor. 10 final policy rollouts (episodes vertically, rollouts horizontally) for most performant method. Task: (Blue, Orange, Green, Red) corridors. Policy distribution over categorical latent space for л and To plotted (between policy rollouts, denoted by Ix)) as vertical histograms, with colour and width denoting category and probability.
E.3 FINAL POLICY ROLLOUTS
In this section, show final policy performance (in terms of episodic rollouts) for APES-HI, across each transfer domain. We additionally display the categorical probability distributions for nh (xx) and =u (F(xx) across each rollout analyse the behaviour of each. F(.) denotes the chosen information gating function for the prior (referred to as IGF(.) in the main text). For CorridorMaze 2 and 4 corridor, seen in Figs. B and (a) we see that the full method successfully solves each respective task, correctly traversing the correct ordering of corridors. The categorical distributions for these domains remain relatively entropic. general, latent categories cluster into those that lead the agent deeper down corridor, and those that return the agent to the hallway. Policy and prior align their categorical distributions in general, as expected. Interestingly, however, the two categorical distributions deviate the most from each-other at the hallway, the bottleneck state (Sutton et al., 1999). where prior multimodality (for hierarchical Tu) exists most (e.g. which corridor to traverse next). In this setting, the policy needs deviate from the multimodal prior, and traverse only the optimal next corridor. We also observe, for the hallway, that the prior allocates one category to each of the five corridors. Such behaviour would not be possible with a flat prior.
Fig. ((b) the same information for Stack 4 blocks. APES-HI successfully solves the transfer task, stacking all blocks according to their masses. Similar categorical latent-space trends exist for this domain as the previous. Most noteworthy is the behaviour of both policy and prior at the bottleneck state, the location above the block stack, where blocks are placed. This location is visited five times within the episode: at the start So, and four more times upon each stacked block. Interestingly, for this state, the prior becomes increasingly less entropic upon each successive visit. This suggests that the prior has learnt that the number of feasible high-level actions (corresponding to which block to stack next), reduces upon each visit, as there remains fewer lighter blocks to stack. that for 5o, the red categorical value is more favoured than the rest. Here, the red categorical value corresponds to moving towards cube O, the heaviest cube. This behaviour is as expected, as during BC, this cube stacked first more often than the others, given its mass. For this domain, akin to Corridor the policy deviates most from the prior at the bottleneck state, as here it needs to behave deterministically (regrading which block to stack next).
also interesting
23

---

Published as a conference paper at ICLR 2023
(F())))
[
(F(zg)
rollout
(a)) CorridorMaze 2 corridor. 10 final policy rollouts (episodes vertically, rollouts horizontally) for most perfor- ((b) Stack 4 blocks. Top) Policy rollouts for most mant method. Task: (Blue, Orange) corridors. Policy dis- tribution over categorical latent space for n and wo' plot-
rollout
performant method. Task: stack blocks in order (0, 1. Bottom) policy distributions akin Fig. ((a) Horizontal dashed lines bottom plots
ted (between policy rollouts, denoted by 2x)) vertical histograms, with colour and width denoting category and probability. Colour here does not correlate to corridor.
refer to current sub-task (block stack), vertically transitioning upon each completion.
Figure Final transfer performance for most performant method. Tasks are solved. Displaying latent distribution for both policy and prior. For both domains, policy deviates most from prior the bottleneck state (hallway/stacking zone, for Maze/tckMaa//Stcckk, where prior multimodality exists (e.g. which corridor/block to stack next), but where determinism is required for the task at hand.
Figure 10: Typical policy rollout, early during training. for APES on Stack blocks. We plot snapshots of single typical policy rollout early during training. Video snapshots unroll left-to-right, top-to-bottom. APES explores at the individual block stacking level.
24


Reviews:
[{"review_id": "yWptuqH3044", "reviewer": "Reviewer_Dj5g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission addresses the problem of information asymmetry (IA) in hierarchical behavior priors as proposed in Tirumala et al. (2020). The role of IA is to limit the prior's access to observations or actions so that the behaviors that are captured are general rather than environment- or instance-specific (for example, restricting observations to proprioceptive sensing for simulated robots might result in the prior capturing general locomotion behavior). As such, the concrete choice of IA is a design decision usually performed by humans, and the main contribution of the paper here is to instead learn this asymmetry in a data-driven manner. The setup that is examined here is that of a high-level policy prior, and IA concerns the amount of past states presented to this prior. The proposed method employs soft attention on the entire history of states for learning IA.", "summary_of_the_review": "This appears to be a good paper addressing a relevant topic. Due to issues wrt clarity and the limited experimental section, I'm leaning slightly in favor of rejection at this point.", "strength_and_weaknesses": "Strengths:\n- Highly relevant topic with focus on a fundamental trade-off when distilling behavior priors.\n- Well-motivated method that effectively automates this trade-off as part of policy training.\n- Strong performance in the tasks considered.\n\nWeaknesses:\n- Information Asymmetry not only concerns the amount of history presented, but also the dimensions of the state that are relevant. It would have been nice to see a further application of attention across state dimensions, e.g., in a simple navigation task. This could then also involve environments for which the selected baselines have been originally developed.\n- On a similar note, the considered environments are relatively simple, although the sparse reward structure renders them of course challenging for an RL algorithm. It would be great to see that the proposed method does not introduce regressions on tasks in which standard behavior priors work well.\n- The presentation and writing is at times unclear and could be improved (see below).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Y1dTyTo_c1", "reviewer": "Reviewer_ia3Q", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets the problem of skill-transfer in reinforcement learning by focusing on learning information asymmetry from data. The key contribution is formalizing the notion of information asymmetry as a unifying perspective on development of skills autonomously, and providing a neat knob that can be turned to tradeoff expressivity of skills with transferability. Experiments on simulated robotics tasks like block stacking and maze navigation show the utility of the proposed framework in skill emergence and transfer. ", "summary_of_the_review": "My main concerns with the paper are minor and relate to proper explanation of some theoretical and empirical results, and some discussions of related works. These can be easily incorporated in the revisions, and as such I am recommending that the paper be accepted. ", "strength_and_weaknesses": "Strengths\n\n1. I found the unifying perspective of Information Asymmetry to be very interesting in connecting skill-learning algorithms that condition on different aspects of the observation space, and in analyzing them under a common umbrella. \n\n2. The experimental settings of maze navigation and block stacking are insightful because the skills in these case are intuitive. Also, the tasks are non-trivial to solve which makes the comparisons with baselines clear - in that the baselines are not able to solve some of the tasks which the proposed algorithm is able to. \n\nWeakness\n\n1. The theoretical sections are a bit hard to follow through. In particular it is not clear whether the expressivity-transferrability tradeoff emerges due to the specific sequential nature of the problem being considered and whether the tradeoff would still exist in the usual multi-task style setup described in section 3.1\n\n2. It is unclear what Figure 6 is trying to show: it seems like skill-level exploration, but without the corresponding visualizations of robot behavior it is unclear whether these diverse trajectories are \"meaningfully diverse\" \n\n3. Some related works in model-based skill-transfer are not dicusssed (see A and B below)\n\nA) Xie, Kevin, Homanga Bharadhwaj, Danijar Hafner, Animesh Garg, and Florian Shkurti. \"Latent Skill Planning for Exploration and Transfer.\" In International Conference on Learning Representations. 2020.\n\nB) Shi, Lucy Xiaoyang, Joseph J. Lim, and Youngwoon Lee. \"Skill-based model-based reinforcement learning.\" arXiv preprint arXiv:2207.07560 (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Xw1GHd2txq", "reviewer": "Reviewer_RbDM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an approach to the problem of transferring learned behaviors between tasks in a sequential setting, called Attentive Priors for Expressive and Transferable Skills (APES). They first present two theorems: a) with more input variables, the shift of the input distributions in terms of KL divergence gets bigger and b) conditioning on more variables leads to better expected knowledge distribution in terms of the KL divergence between output distributions, which together suggest an expressivity-transferability trade-off. The authors then suggest APES, which learns soft masks for input to each module in the KL-regularized hierarchical policies. They test and perform analyses of their approach in the CorridorMaze and Stack environments to show the empirical effectiveness of the approach.", "summary_of_the_review": "I find the connection between the theorems on the expressivity-transferability trade-off and the proposed method somewhat interesting, but my major concern is that the connection may not hold in the case of the soft mask learning.", "strength_and_weaknesses": "Strengths\n\n- I like the presentation of the expressivity-transferability trade-off using the theorems in terms of KL divergence, and the attempt to connect them with the proposed method.\n- They analyze the experimental results in multiple ways, which could help understanding the empirical effectiveness of the proposed approach.\n\nWeaknesses\n\n- I'm concerned that by using the soft masks to allow them to be trained, the method could lose most of its theoretical connection with Thm.3.2 and thus the expressivity-transferability trade-off. It is because, unless some mask is strictly zero (or near-zero in practice) the information the corresponding variable carries does not go away. This is probably supported by Fig.5, where the minimum mask value is $\\approx \\exp(-3.1) \\approx 0.045$. Their corresponding variables may not play big roles in the prediction, but it's hard to say that they are completely ignored.\n- The scales of the soft masks can largely be affected by the scales of the input variables. I'm curious if they are properly normalized in the experiments.\n- The writing could be improved in some ways. In Eq.3, $\\pi$ and $\\pi_0$ are not conditioned on the masks and $\\mathcal{O}_{\\text{APES}}$ does not take the masks as its input. The boldfacing of the inputs in the text and Thm.3.2 do not match. The second sentence of Sec.2 is missing an \"and\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7nnlBzKsWt", "reviewer": "Reviewer_pBgA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the problem of efficient transfer by combining two methods in the literature: hierarchical reinforcement learning (HRL) and KL-regularized RL. The paper argues that a key element to the transferability of the policies is by considering information asymmetry between them. To achieve this goal, the authors propose to employ soft attention on the inputs and learn this attention mask through an entropy objective on its weights (equation 9). To motivate information asymmetry, the authors two theorems on covariate shift and expectation distillation. The paper performs experiments on two domains, one a toy task and a harder stacking task, and report improved performance compared to other baselines.", "summary_of_the_review": "The paper proposes an interesting combination between HRL and KL-regularized RL and motivate the idea of information asymmetry as a way to better transfer across tasks. The paper relates the idea of information asymmetry to transferability, which is not always true. Moreover, transferability is related to covariate, which once again is not always true. More nuances are needed. For the experiments, the setting seems more complex than it should and is only evaluated with a few seeds.", "strength_and_weaknesses": "# Strengths \n- The paper tackles an important problem by combining two existing approaches in an interesting fashion\n- The paper compares with a large set of baselines\n- The theorems presented could be of interest to the community\n\n# Weaknesses\n- The claims do not seem necessarily true. In particular, the authors claim that successful transfer relies on information asymmetry, which may or may not be true, depending on the method one considers for transfer learning.\n- The relation between the theorems and the ability to transfer is not necessarily direct. Some nuances would be important here\n- The empirical setup is quite complex and the number of seeds is very low, bringing into question the reliability of the results.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster