Disentangled Cumulants Help Successor Representations Transfer to New Tasks
Authors: Chris Grimm
Irina Higgins
Andre Barreto
Denis Teplyashin
Markus Wulfmeier
Tim Hertweck
Raia Hadsell
Satinder Singh
Venue:  (2020)
Keywords: reinforcement learning
representation learning
intrinsic reward
intrinsic control
endogenous
generalized policy improvement
successor features
variational
monet
disentangled

Abstract: Biological intelligence can learn to solve many diverse tasks in a data efficient manner by re-using basic knowledge and skills from one task to another. Furthermore, many of such skills are acquired through something called latent learning, where no explicit supervision for skill acquisition is provided. This is in contrast to the state-of-the-art reinforcement learning agents, which typically start learning each new task from scratch and struggle with knowledge transfer. In this paper we propose a principled way to learn and recombine a basis set of policies, which comes with certain guarantees on the coverage of the final task space. In particular, we construct a learning pipeline where an agent invests time to learn to perform intrinsically generated, goal-based tasks, and subsequently leverages this experience to quickly achieve a high level of performance on externally specified, often significantly more complex tasks through generalised policy improvement. We demonstrate both theoretically and empirically that such goal-based intrinsic tasks produce more transferable policies when the goals are specified in a space that exhibits a form of disentanglement. 

Under review as conference paper at ICLR 2020
DISENTANGLED CUMULANTS HELP SUCCESSOR REPRESENTATIONS TRANSFER TO NEW TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
Biological intelligence can learn to solve many diverse tasks in a data efficient man- ner by re-using basic knowledge and skills from one task to another. Furthermore, many of such skills are acquired without explicit supervision in an intrinsically driven fashion. This is in contrast to the sfate-of-thh-ar reinforcement learning agents, which typically start learning each new task from scratch and struggle with knowledge transfer. In this paper we propose a principled way to learn a basis set of policies, which, when recombined through generalised policy improvement, come with guarantees on the coverage of the final task space. In particular, we concentrate on solving goal-based downstream tasks where the execution order of actions is not important. We demonstrate both theoretically and empirically that learning small number of policies that reach intrinsically specified goal regions in a disentangled latent space can be re-used to quickly achieve a high level of performance on an exponentially larger number of externally specified, often significantly more complex downstream tasks. Our learning pipeline consists of two stages, First, the agent learns to perform intrinsically generated, goal-based tasks in the total absence of environmental rewards. Second, the agent leverages this experience to quickly achieve a high level of performance on numerous diverse externally specified tasks.
INTRODUCTION
Natural intelligence is able to solve many diverse tasks by transferring knowledge and skills from one task another. For example, by knowing about objects and how move them in 3D) space, it is possible to learn how to sort them by shape or colour faster. However, many of the current state-of-the-art artificial reinforcement learning (RL) agents often struggle with such basic skill transfer. They are able to solve single tasks well, often beyond the ability of any natural intelligence (Silver et al.] 2016 Mnih al. 2015 Jaderberg et al. (2077), however even small deviations from the task that the agent was trained on can result in catastrophic failures (Lake et al. 2016 Rusu et al.) 2016). Although improving transfer RL agents is an active area of research (Higgins et al. 2017a
Rusu et 20166 Nair et al. 2018 Barreto et al. 2018 Wulfmeier et al. 2019 Torrey & Shavlik
Caruana [199]] et a 2017] Riedmiller ct al. 2018), most typical deep RL agents start learning every task from scratch. This means that
2010 Taylor & Stone, 2009 Thrun & Pratt 2012
each time they have to re-learn how to perceive the world (the mapping from a high-dimensione
observation to state), and also how act (the mapping from state to action), with the majority of time arguably spent on the former. The optimisation procedure naturally discards information that is irrelevant the task, which means that the learnt state representation often unsuitable for new tasks. Biological intelligence appears to operate differently. A lot of knowledge tends to be discovered and learnt without explicit supervision (Tolman 1948 Clark 2013 Friston| 2010). This basic knowledge can then form the behavioural basis that can be used to solve new tasks faster. In this paper we argue that such transferable knowledge and skills should be acquired in artificial agents too. In particular, we want to start by building agents that have the ability to discover stable entities that make up the world and to learn basic skills to manipulate these entities. Compositional re-use of such skills enables biological intelligence to find reasonable solutions to many naturally occurring tasks, from goal-directed movement (controlling your own position), to food gathering (controlling the position of fruit and and nuts), building simple defence system (re- tre-poitiiniig multiple stones into a fence or digging a ditch). In this paper we concentrate on goal- natural tasks that can be expressed in natural language, and that do not require a specific execution order of actions.

---

Under review as conference paper at ICLR 2020
MDP induced by disentangled state
Idealised world state
Noisy observations
Position y
Position
colour
Colour
W
position y
(x=5, y=6, c=2)
position
Figure 1: The idealised world state completely described by compositions of the following inde- pendent transformations: changes in position x, y and colour. Such a state may be projected into a high-dimensional observation, which may contain lot of irrelevant detail, like the particular instan- tiation of the Qhert, or the grassy background. Disentangled representations recover the meaningful information about the independently transformable aspects of the world and disregard the irrelevant details.
To this end, we propose principled way to learn a small set of policies which can be re-used by the agents quickly produce reasonable performance on an exponentially large set of goal-driven tasks within an environment. We propose method on how to discover these policies in the absence of external supervision, where the agent accumulates a transferable set of basic skills through intrinsi- cally motivated interactions with the environment. This first stage of free play builds the foundation to later solve many diverse extrinsically specified downstream tasks. We suggest formalising such two-stage pipeline as the endogenous reinforcement learning (ERL) setting, in order to provide a consistent evaluation framework for some of the existing and future work on building RL agents with intrinsic learning signals (Gregor et al. 2017: Eysenbach et al. 2019 Hansen al. 2019 Nair et al. 2018 I Laversanne- et al. 2018). We propose a disjoint two step research pipeline, where the agent allowed unlimited access to the environment in the ERL stage, where no extrinsic rewards are provided and the agent is supposed to learn as much as it can through endogenously (intrinsically) driven interactions with the environment. This is followed by a standard RL stage where the success of the previous step is evaluation in terms of data- lata-efficiency of learning multiple diverse extrinsically (exogenously) specified downstream tasks in the same environment. We hope that by working in this extreme two stage setting, where the agents have to learn useful knowledge with no access to task rewards, we can develop algorithms that learn more robust and transferable policies even in the traditional RI. setting.
In this paper we propose to use the ERL stage to discover disentangled features through task-free interactions with the environment, and then solving number of goal-driven self-generated tasks specified in the learnt disentangled feature space. In particular, we suggest learning disentangled features, discretising them into bins each, and learning km feature control policies that achieve the respective bin value of the given feature. We then ce-ombiee the feature control policies learnt in the ERI. solve downstream tasks in the RL stage in few-shot manner using Generalized Policy Improvement (GPI) (Barreto et al. 2018). We demonstrate theoretically and empirically that our proposed set of basis policies that learn to control disentangled features produce significantly better generalisation over a large number of downstream tasks.
Intuitively, disentangled representations consist of the smallest set of features that represent those aspects of the world state that are independently affected by natural transformations and together explain the most of the variance observed an environment (Higgins al. 2018) (see Fig. D Dis- entangled features, therefore, carve the world at its joints and provide parsimonious representation of the world state that also points to which aspects of the world are stable, and which can in principle be transformed independently of each other. We conjecture that disentangled features align well with the idealised state space which natural tasks are defined. Hence, by learning a set of policies that can control these features an agent will acquire a set of basis policies which spans large set of natural tasks defined in such an environment. Note that both disentangled features and their respective control policies can be learnt without an externally specified task, purely in the ERL setting. We provide both theoretical justification for this setup. as well as experimental illustrations of the benefit of disentangled representations in large set of tasks of varying difficulty.
2

---

Under review as conference paper at ICLR 2020
Hence, the main contribution of this work is theoretical result that extends the GPI framework to guarantee achievability on a large set of natural goal-driven tasks given a small set of basis policies that control disentangled features. In particular, discretised into m bins, we can guarantee achievability with a deterministic policy on at least (m++1 1)* downstream tasks by using GPI to recombine km feature control policies discovered and learnt purely through intrinsically driven interactions with the environment in the total absence of environment rewards. Our result holds for any tasks that can be specified in natural language and do not require a particular ordering of the actions to be solved. For example, our approach would be able to solve a task that requires sorting objects in space based on their colour or shape, or tidying up a messy playroom by putting all the toys in box, but it will not be able to solve task like cooking a meal, where the execution order of the different stages in the recipe matters.
demonstrate that given l disentangled features
Related work Past work supervised and reinforcement learning has demonstrated how multi- task learning. transfer and adaptation can provide strong performance gains across various domains (Caruana] 1997| Thrun & Prati [2012] Yosinski et al. 2014 Girshick al. 2014 Jaderberg al., 2017 Riedmiller et al.] 2018] Wulfmeier et al. 2019)) Typically these approaches use hand-crafted auxilarr tasks to boost learning of the downstream tasks of interest, which is not scalable and comes with no guarantees on which set of auxiliary tasks is optimal for boosting performance on a large number of downstream tasks. A number of other past approaches shared our motivation of replacing the hand-crafted auxiliary task specification by an automatic way of discovering a diverse and useful set of policies in the absence of externally specified tasks. The predominant approach so far has been to optimise objective that encourages behaviours that are both diverse and distinguishable from each other (Gregor et al. 2017 Eysenbach et al. 2019, Hansen et al. 2019), or to learn how to solve intrinsic tasks sampled from a learnt representation space (Nair et al. [2018 Laversanne-Fing Finot et al. 2018). While these approaches have been shown to be successful on transterring the learnt policies to solve certain downstream tasks, none of them provided theoretical guarantees the downstream task coverage by the basis set of policies. Such guarantees were however provided by yan Nickerk et al.] (2018) and Barreto et al. (2017, 2018). These papers calculated how well a given set of policies can be transferred to solve a wide range of downstream tasks. However, they left the question of how to discover such a set of basis policies open. Hence, our work provides a unique perspective by addressing both the questions of what makes a good basis set of policies to get certain guarantees on final task coverage, and how these policies may be learnt in the ERL setting. Other related literature worth noting is the work by [Hggins et al. (2017b), who showed that learning a downstream task policy over disentangled representations improved its robustness to visual changes in the environment. Another piece of work (Machado et al. 2018)) demonstrated the usefulness of discovering reward-agnoste options through successor feature learning for improving data efficiency in downstream task learning. The benefits of these options, however, were primarily through improving exploration. No guarantees were given in terms of downstream task coverage.
2 BACKGROUND
Basic Reinforcement Learning (RL) formalism. An RL agent interacts with its environment through a sequence of actions in such way as maximise the expected cumulative discounted rewards (Sutton & Barto 1998). The RL problem is typically expressed using the formalism of Markov Decision Processes (MDPs) (Puterman 1994p. An MDP is a tuple M =
where S and A are the sets of states and actions, P is the transition probability that predicts the distribution over next states given the current state and action s' ~ ((|ss,a), ) R. the distribution of rewards r ~ R(s,a,s') received for making the transition s s s', and y E [0.1) is the discount factor used make future rewards progressively less valuable. Given an MDP, the goal of the agent is to maximise the expected return G,= x - 7 i- This is done by learning a policy π (als)) that selects the optimal action a E A each state s E S. A typical RL problem attempts to find the optimal policy o = argmax
where the expectation is taken over all possible interaction sequences E Er207/r
of the agent's 's policy with the environment. The optimal policy is learnt with respect to particular task operationalised through the choice of the reward function R(s,a,s').
Successor Features The successor feature (SF) representation is a way of decoupling the dynamics of an environment from its reward function. This is done by representing an environment reward
3

---

Under review conference paper at ICLR 2020
Step Disentangled feature discovery
Step 2. Learning fealure control policies
Step 3. Few-shol task learning
ERL
readiing feltari
Figure 2: Schematic illustration of the three steps of our method. First we use an existing method for unsupervised disentangled feature discovery from observations obtained using an exploration policy. We then learn control policies that learn to achieve certain uniformly spread values for the learnt features. Finally, we use the feature control policies to solve tasks using the GPT framework. The first two steps do not require any extrinsic rewards.
as r(s, s') ( T representation does not decrease the expressivity of since assumptions are made the form of 6. Moreover, this representation allows for decomposing value functions as follows:
where ((s,a,s') is a vector of environment features. Notably, this
("(saa==E"
wj - (=,0(s wy.
where (s.a)) is vector of reward-innde -eward-indeeendentt successor featuress.
GPI GPE Generalised Policy Improvement (GPI) and Generalised Policy Evaluation (GPE) [Brrrtto al. 2017) can be used to transfer set of existing policies to solve new tasks. The framework is specified for a set of MDPs:
(2)
induced by all possible choices of weights w that specify all possible rewards r, given a state space S. action space A, transition probabilities P, discount factor and features (ss,a,s'). Note that the features are meant to be the same for all MDPs M E A Given policy x; learnt to solve task i specified by w;, evaluate its value under different reward = "(s,a,s')) ww using GPE:
(=,aa)=((ssa))) w
using definition of successor features defined in (I]
Hence, given a set of policies Ti, T2. n. induced by rewards T1, T2. r, over a subset of the get policy πy for task induced by rj (note that Mj E Mo
MDPs M' c M My according to:
(5(3)) argmax max (888a) w.
(4)
ENDOGENOUS RL WITH GPE AND GPI
This section provides general overview of the proposed ERL pipeline. consisting of (1) a represen- tation learning phase, (2) an intrinsic reinforcement learning phase and (3) few-shot learning phase when new task is presented, where steps 1-2 form the ERL stage, and step 3 forms the RL stage (see Fig.2) (2. In Sec. 4 we will present the main theoretical contribution of our paper where we will discuss why particular version of the pipeline that uses disentangled features expected to perform well in the RL. stage.
Representation learning. At the beginning of the pipeline our agents learn a parameterized representation of the environment's state which to define of features 61 (s) E R
4

---

Under review as conference paper at ICLR 2020
Each feature & will be used cumulant that gives rise to RL task. Specifically, each feature & will induce an associated optimal policy n, that maximises the expected discounted sum of ø;.
We propose defining features Φ; in the following way. Let fL* S + [0, 1Jth be arbitrary continuous function. We define a discretization of each fi(s) into m uniformly spaced bins: [0.1/m), /mm.
1)/m,1), denoted bı....mm respectively. Using this notation,
define a set of n n= -mk features as follows:
(5) where [(-} is the indicator function. Note that, to facilitate the exposition, we use two indices to refer specific feature: obviously, we can "flatten" these indices and treat the features as a vector if needed. Intuitively, we can think of the task induced by diy (s,a) as setting the i-th representational latent dimension a value in the interval by. Specific choices of m and in our experiments are explained in the supplementary material.
(sy(s,a) = ([((s))
As discussed above, each feature øij gives rise an RL task. The second stage of our pipeline consists of solving these tasks. Crucially, instead of computing the value function of policy /i with respect to cumulant Øi only, we will compute the successor features of x with respect to all cumulants:
Intrinsic RL
Vth (6,0)=Bi Ens
r hou (st,0:)|s0 8,00
(6)
where (s) is one of the optimal policies induced by cumulant Øij. that is, (s)(s)
argmax, Q7, (s,)()))
Collectively the successor features 1i (s.a) can be thought as an n x n matrix v with cumulants in one dimension and policies in the other dimension. The matrix V represents the agent's 's knowledge of how to manipulate the features of the environment. We will also use our double-suhscript notation
to refer to specific elements L (s.a): V (1n). (s,a)= (s,a).
Few-shot learning phase Provided our agent has invested an initial effort to accurately learn the matrix V, we can now leverage it to perform few-shot learning. First, note that any linear combination of cumulants di:n ow E, w.db, is itself a cumulant, We can then define the set of cumulants
diw (8,a)
Σ Cusybeyyssa /w TR kxm 2.
Given an arbitrary task, we can find cumulant &w E @ that approximates the task well as possible. One way to do so is to select w E kxmm such that
W = argmin E (s,a)>D
(ssa)
(8)
w/ w'ERex
where D is distribution over S X A and ||- is norm. Note that g is linear regression.
Once we have computed w & we can use the successor features of policy mi to evaluate it on task dw (a process sometimes referred to as Generalized Policy Evaluation or GPE);
(s.a) Σ
wunhi (s,a),
I.A
where Q (s.a) is the action-valee function of mi under cumulant dw. Finally, by using GPI we can directly compute policy based on the known policies n'y;
GPI (s) = argmax max Th (s,a).
(10)
While dw is not guaranteed to be optimal with respect to dw. its performance on this task least as good, and generally better, than that of the known policies Tī (Barreto et al. (017)) Moreover, the computation of GPT is immediate given the matrix & and Since assume the matrix V has been pre-computed the ERL phase, this essentially reduces an RL problem to the regression problem shown in (
5

---

Under review as conference paper at ICLR 2020
4 THEORETICAL RESULTS
This section forms the main theoretical contribution of this paper. We highlight the importance of the choice of latent representation used in our learning pipeline. Particularly, we show that when the agent's latent representation exhibits a particular form of disentanglement, we can exploit this property to both accelerate the learning of our successor feature matrix and guarantee that GPI finds solutions to certain families of tasks.
Disentangled representations have recently been connected to symmetry transformations (Higgins al. 2018), a powerful idea borrowed from physics. Roughly speaking, symmetry transformation for a system is a transformation that leaves some property of the system unchanged. In the context of any structure preserving transformation,
machine learning, the idea of symmetries is generalised like a change in colour, shape, or position of an object. All of these transformations commute with each other in the natural world and can be applied independently, hence forming a symmetry group. This induces an MDP in the state space of discretised disentangled latent dimensions that resembles a hypercube (see Fig. D where the nodes on each edge correspond to different values of a single disentangled dimension. Due to the commutative property of disentangled transformations, it suffices to learn how to move along the individual edges of such a hypercube be able to reach any state inside the hypercube. Hence, by learning disentangled latent space with z dimensions, and discretising them into bins, we can re-use 2b feature control policies to solve (b + 1)a downstream goal-based tasks, where b+1 1 includes the extra bin value per disentangled dimension that indicates that the value of the corresponding dimension is irrelevant to the task. This intuition is discussed more formally next.
Optimal independent controllability Here we formalise a disentangled representation in terms of features that can be optimally controlled without affecting other features. Let Φi.n be set of features and let ग. be the optimal deterministic policy associated with the control task induced by the cumulant 1 (s(ss) (s) E ]. with R, C R. Let (st) be a sequence of random variables generated by starting in state So following n.- We call 01 optimally independently controllable (OIC) if [(d)(s))] ( (8o) for all j and E (1.....N).
Note that if we use the features defined in (5) can have control tasks induced by (() = } {{{(()) E ト for j 1, In this case two features dig and Φn can be OIC only if / h. We will abuse the terminology slightly and say that f are OIC if any pair of the induced features &ey.dhs is OIC when i * h. Without loss of generality we will assume henceforth that we are using features defined as in (5).
An immediate consequence of a set of OIC features is that values under rewards and policies associated with different features have simple form:
Lemma 4.1. When fim are optimally independently obtainable the successor feature matrix, V, has the following form:
(s,()
iiit
W (0,,,
(11)
v th (s,a)
else.
Proof. The proof follows directly from the definition of OIC features. If 1 +f i then under क|. fi(st)= = fi(so), and thus Φn (8t,0r) = 6rn (s0,0t), giving:
00
Vi (s.a)=I E E
7 You (s)rae)
=
E
Oun don (0. =..00
Sen (8,a).
(12)
Intuitively, Lemma 4.1 states that if the feature f. associated with policy ti is different from the cor-
responding feature fi used to define Φu the associated successor feature V (1))) (s,a) w (s,a)
reduces to (1 (1-7)--ob (8,a). This reduces learning the matrix V learning subset of its entries.

---

Under review as conference paper at ICLR 2020
Guarantees for conjunctions of goal-based tasks In addition to simplifying the process of learn- ing the successor feature matrix V, features with the OIC property come with guarantees under GPI for certain goal-based tasks. We define a goal- boal-bbsed task as one whose reward function has the form
Rc = 1{$EG]
(13)
where G C.S. Given the above definition, we say that a policy x "achieves" G if V (8) > 0 for all S
Our uniform discretization of features fi.k into bins bim naturally induces partition over of the state-space:
B(S) - (Bi i {(m]]
(14)
where
B 11, ==1
(15)
Intuitively, we can think of each partition B one of the possible m configurations of the features Φi (note that there are fewer than 2 configurations because some of them are impossible, as two bins associated with the same feature cannot be active at the same time). We can then think of these partitions as goal regions analogous to (3). We now show that for any goal g (SS) there exist a linear combination of the cumulants the that leads to GPI policy that achieves g.
Theorem 4.1. If fis are OIC, then for any E B(S) there exists w E kxm such that GPI as defined in and (10) achieves g. One such w is given by wfy = =b,()))bbb
Proof. Recall that π GPI argmax, Q (s,a), where (s,a) max.s En with v Vi (s,a). We
begin by rearranging terms in Qwa:
Q max (s,a) = max
wit in (s.a)
1h
m
37
max
ih ih
V
(s,0)
Wit Wi (s,a)
1= Lipi
max
W
sh ih
Joj
(s.a)
in Sur (s,a)
(16)
h=1
m
m
max
V T ih ih
(8,a)
wi wi. (8,a)
1-7 w (er(()))))
C(s)+man max
Wihh
v
(5,a)
Din (s,a)
where the third equality follows from Lemma 4.1 and C(s) captures du (s.a) terms (which do not depend on a or i).
First note that, from the form of w9, for each there is exactly one j such that w/t = with all other entries being O. Denote this j as b(t). We can then rewrite:
Q max (s,a) = (s)+ max if
v (b(()) (s,a)
1-> (s,a
(17)
= C(ss) + max
v
TiMae (b(5))
(s,a)
Dio(i) (s,a)
Next notice that 20)
Face
(8,a)
Tite 1- Pao (8.n) 0s f(()) boca) and s0(s) (s,a) otherwise, giving:
Qmay (8,a)= C(s)+ max v
ib(1) (s,a)
(18)
.EW(s)

---

Under review as conference paper at ICLR 2020
where ((s) (i: & bo(e)} or, more plainly, the set of feature indices that have not been achieved yet. This gives
GPT (8) (6)==rgmaa Quaa (8,0) = argmax max Vi Tinc .EW(s)
(s,a), <b(i)
(19)
a
implying that Th will persistently pursue the "unachieved" feature (0th = 0) that is easiest to "achieve" (that is, to be set to 6iL_ = 1) among the features associated with nonzero elements in wa. This means that eventually all such features will be set to 1, which in turn implies that goal g has been achieved.
5 SPRITEWORLD EXPERIMENTS
In this section we experimentally validate that an agent can effectively use task-free interactions with an environment to gain boost in data efficiency across wide range of subsequent tasks. In particular, we test whether an agent that uses GPI to transfer set of feature control policies discovered in the ERL setting has boost in performance over a baseline DQN agent that learns each task from scratch. We also validate whether disentangled feature control policies form a better basis for transfer compared to entangled feature control policies, and whether our pipeline outperforms DIAYN (Eysenbach et al., 2019) a state of the art approach for discovering a diverse set of policies the absence of external tasks. We chose DIAYN as our baseline, since it is the closest method to ours in terms of motivation (c.g. it does not rely on conl-conditioning in the RL stage unlike Nair et al. (2018). ), and is the current published sfate-of-the- oh-the--ttrtrr- art method in its class (unlike Breker et al. (201 7 Hansen et al. (2099) )
We validate our ideas on a toy Spriteworld environment Watters et al. 2019) (see Fig.B), The environment contains agent and two sprites. The action space is 8-dimensional and consists of moving the agent up/down or left/right, as well as the same four actions but for dragging objects. It is only possible to drag objects if the agent is standing on them. Furthermore, dragging ac- tions move the agent slower than the standard move actions. This environment makes it easy to define a wide range of diverse natural tasks of different diffi- culty level that can be easily expressed through language. In particular, here we concentrate on set of navigation based tasks, which specify a goal location for the agent or the objects. The easiest tasks are specified in terms of the final position of the agent ("top", "middle", "bottom" horizontally; and "left", "cen- tre" or "right" vertically). The harder tasks make the locations more specific (by specifying both the horizontal and vertical coordinates, e.g. "top left" "bottom right"). We also specify equivalent tasks but in terms of the final ob- ject position (c.g. "sqquare at the bottom" an easy object task, or "circle at the top right" as a hard object task). Note that the object-based tasks are more dif- ficult than the agent-based tasks, because the agent position can be controlled
agent
Figure Spriteworld en- vironment. The agent can move up/down. left/right and drag objects when it steps on them.
3:
directly in the action space, while controlling the object position requires more elaborate policies that are also dependent on the agent position. We also specify tasks in terms of disjunctions of agent or ob- ject goal locations (e.g. easy tasks like "agent to the left OR square to the top" or hard tasks like "agent to the top left OR square to the top right"). Finally, our hardest tasks are specified as conjunctions of specific locations for both objects (e.g. "square to the top left AND circle to the bottom right"). Note that the goal locations of the downstream tasks may not directly related to binning specified during the ERI. feature control policy learning. Indeed, GPT should be able to generalise the given set of fea- ture control policies to solve a wider range of tasks spanned by the bin boundaries. In the Spriteworld environment the agent receives a reward of if the relevant aspects of the environment state are within their respective goal locations, otherwise the reward is O. Each episode terminates immediately if the goal is achieved. The agent and the objects are initialised in random positions sampled uniformly within the environment at the start of each episode. We evaluate the performance of our agent and the baselines tasks sampled from each of the 7 different task classes. Given the structure of our tasks, the average reward corresponds to the average number of episodes on which the agent solves the task.
As illustrated by Lemma 4.1 when our features Φin are OIC, the matrix of values (s.a) takes on a specific form where many off- diagonal entries are completely determined by 6i.n (s,a) and y. By assuming that disentangled representation has the OIC property we can replace these off off-diagonal entries by their exact values, both reducing the cost of computing v and associated
Off-Diagonal Trick
8

---

Under review as conference paper at ICLR 2020
Agent
Object
easy
easy
245
+20
024
0.2
92-
0.4
34
33
30
Hap Inir
stap 2n01
Disjunction
Conjunction
casy
hard
Disent biik
Disent
Ent
DWAYN
455
Hep 1m00
210 070
1ep 1m:
Figure 4: Average reward achieved by the different methods across the same set of tasks. The agent receives a reward of 1 if the goal location reached, otherwise receives 0. The rewards are averaged across 3 seeds and 3 tasks per task category. Agent tasks include moving the agent to specified position. Easy tasks specify a vertical or a horizontal position (e.g. "get the agent to the top"). Hard tasks specify conjunction of vertical and horizontal position (e.g. "get the agent to the bottom right"). Object tasks are similar to agent tasks but specify the goal position of one of the objects (e.g. "get the square to the centre left"). Disjunction tasks are set by specifying goal in terms of disjunction of the corresponding easy or hard agent and object tasks (e.g. "get the agent to the left OR get the circle to the middle"). Finally, conjunction tasks are specified as a particular vertical and horizontal position for both objects (e.g. "get the square to the top left AND the circle to the bottom right"). GPT with disentangled led features and the off-diagonal trick is able solve all the tasks least 50% of the time almost immediately. DQN takes orders of magnitude more steps to achieve similar performance. DIAYN has discovered some agent control policies that allow it to solve some agent based tasks, but it never discovered how to control objects. We used 9 bin feature discretisation to train the feature control policies used by GPI.
error of estimating these off-diagonal terms. We denote the setting of all off-diagonal terms with index (1h). to (1
(s,a) as the "off-diagonal trick."
Results We evaluated how well our approach works in the scenario where disentangled features are the true x and y positions of the agent and the objects, and the entangled features are rotations of the disentangled features. Fig. 4 demonstrates that GPI over feature control policies provides almost immediate boost in performance over the DQN baseline. This effect gets more prominent as the task difficulty increases, whereby the number of steps before DQN reaches the same performance as the GPI agent increases with task difficulty. The GPI agent is able to solve the tasks most of the time within around 50k learning steps, while the DQN baseline typically requires 400k steps. We also see that GPI over disentangled features provides a significantly bigger jump in performance compared to GPI over entangled features. Finally. it is clear that the "off-diagonal trick works well for the disentangled GPI, which suggests an additional benefit of better computational efficiency during FRI. learning. Moreover our comparisons against DIAYN, we found that the competing method could only learn perform tasks involving the agent, failing to learn to interact with the objects. Finally, Fig. Jeemonttttss that our results are relatively robust over the choice of the disentangled feature bin number (m (5,7,9)) and whether the task goal regions are set to the ERL bins exactly (aligned in Fig.51 5) or not.
jiggerjump
Note that although we have tested our approach simple domain using ground truth features, our theoretical results should hold for any complex natural environment and for any natural tasks that can be specified in terms of disentangled transformations and that do not rely ordered execution. This

---

Under review as conference paper at ICLR 2020
Agent
Object
easy
easy
20
04
02
201 IMX
0.5
Disjunction
Conjunction
essy
Bn -8
Bn - aigrd
180
12
05
Figure 5: Average reward achieved by GPI with disentangled features and the off-diagonal trick on the same tasks as in Fig. but using different numbers of feature discretisation bins. The performance of approach decreases with smaller numbers of bins, since GPI has fewer policies to work with, but still significantly outperforms all baselines shown in Fig. in terms of data efficiency. Aligning the downstream tasks with the bin discretisation used to train the feature control policies (bin=7, aligned) slightly improves the performance of our approach, especially on harder tasks.
because disentangled representation learning aims find low-dimensional representation that is equivariant with respect to symmetry transformations, which are a ubiquitous property of our world (Livio 2012). Hence, any arbitrarily complex natural environment should be possible describe in terms of a relatively small number of disentangled dimensions. We leave empirical demonstrations on more complex environments using learnt disentangled features to future work.
CONCLUSIONS
We have proposed principled way learn and recombine basis set of policies that guarantees achievability for a large set of natural tasks that do not require a particular execution order of actions. We have demonstrated that these policies access to external rewards and has to learn through intrinsically driven interactions with the environ- ment. We theoretically justified and experimentally verified three-stage pipeline, where the agent 1) discovers disentangled features, 2) learns set of basis policies through specifying intrinsic rewards when a particular feature achieves particular range of values, and 3) uses GPI over these policies to bootstrap reasonable performance and which do not rely on a particular execution ordering. We have empirically demonstrated that GPI over disentangled feature control policies produces better task coverage and learning efficiency simple Spriteworld domain compared to a baseline DQN agent and DIAYN, a state-of saaee--f-theaaaee the-hh-a art method for discovering diverse policies useful for downstream tasks through task-free interactions with the environment. In the future work we would like to generalise our approach to tasks that require a particular order of action execution, well continuous action spaces.
be learnt in the ERL setting, where the agent has
wide range of natural tasks which can be expressed in language
REFERENCES
Andre Barreto, Will Dabney, Remi Munos, Jonathan Hunt, Tom Schaul, Hado Hasselt, and David Silver. Successor features for transfer in reinforcement learning. NIPS, 2017.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using
10

---

Under review as conference paper at ICLR 2020
successor features and generalised policy improvement. NIPS, 2018.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
A. Clark. Whatever next? predictive brains, situated agents, and the future of cognitive science. W Behavioral and Brain Sciences, 36, 2013.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: learning skills without reward function. ICLR, 2019.
Karl Friston. The free-energy principle: unified brain theory? Nature Reviews Neuroscience, 2010.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEK conference on computer vision and pattern recognition, pp. 580-587, 2014.
Karol Gregor, Danilo Rezende, and Daan Wierstra. Variational intrinsic control. JCLR, 2017.
Steven Hansen, Will Dabney, André Barreto, Tom Van de Wiele, David Warde-Farley. and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arxiv, 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning basic visual concepts with a con- strained variational framework. ICLR, 2017a.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. DARLA: Improving zero-shot transfer in reinforcement learning. ICML, 2017b.
Irina Higgins, David Amos, David Pfau, Sehastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards definition of disentangled representations, arXiv, 2018.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. ICLR, 2017.
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, pp. 1-101, 2016.
Adrien Laversanne-Fing Finot, Alexandre Péré, and Pierre- Yves Oudeyer. Curiosity driven exploration of learned disentangled goal spaces. arxiv, 2018.
Mario Livio. Why symmetry matters. Nature, 490:472-473, 2012.
Marlos C. Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. ICLR, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David S Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. 2015.
Nature, 518774),
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. arxiv, 2018.
M. L. Puterman. Markov Decision Processes Discrete Stochastic Dynamic Programming. John Wiley Sons, Inc., 1994.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Tobias Springenberg. Learning by playing solv- ing sparse reward tasks from scratch. ICML, 2018
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick. Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arxiv, 2016.

---

Under review as conference paper at ICLR 2020
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner. Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
Matthew Taylor and Peter Stone. Transfer learning for reinforcement learning domains: survey. Journal of Machine Learning Research, 100u)):1633-1685, 2009.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Edward C. Tolman. Cognitive maps in rats and men. The Psychological Review, 55(4):189-2)) 1948.
Lisa Torrey and Jude Shavlik. Transfer learning. In Handbook of research on machine learning applications and trends: algorithms, methods, and techniques, pp. 242-264. IGI Global, 2010.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-
learning. In Thirtieth AAAI conference artificial intelligence, 2016.
Benjamin van Niekerk, Steven James, Adam Earle, and Benjamin Rosman. Composing value func- tions in reinforcement learning. NeurIPS, 2018.
Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Livescu. Deep variational canonical correlation analysis. arXiv preprint 161.03444,, 2016.
Nicholas Watters, Loic Matthey, Sebastian Borgeaud, Rishabh Kabra, and Alexander Ler- chner. Spriteworld: A flexible, configurable reinforcement learning environment. 2019.. URL BRL/tttps///iitaut wwt. deeppind/
spritewordd/
Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg, Michael Neunert, Tim Hertweck, Thomas Lampe. Noah Siegel, Nicolas Heess, and Martin Riedmiller. Regularized hierarchical poliiciss for compositional transfer in robotics. arXiv arXiv: 1et..1906...... 2019.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pp. 3320-3328, 2014.
12

---

Under review as conference paper at ICLR 2020
A
APPENDIX
A.1 SPRITEWORLD ENVIRONMENT
The Spriteworld environment consists of room without obstacles with agent and two objects: circle and square. The agent can take 8 agents consisting of regular movements (up, down, left, right) and 4 "dragging movements" which mirror the regular movements but move the agent half as far. Objects in the environment can overlap and pass through each other. When the agent overlaps with an object and executes a dragging movement, both the agent and the object are moved together. For our experiments we use a vectorized version of our environment state as observations to our models, consisting of scalars representing the the and y positions of the agent and each object. When the environment is reset, both the agent and objects are randomly positioned.
A.2 LEARNING THE SUCCESSOR FEATURE MATRIX
In this section we describe the procedure for learning approximation of our successor feature matrix y whose entries are defined by (6). It helpful visualize this matrix as follows:
V (s,a)= T (s.a)
i (s,a)
T (s.a)]
where each th corresponds to the full vector of successor features evaluated under policy mīy: (s, = Wi (s,a)
Analogously, we also define the column vector of corresponding cumulants as ((s.a) (s,a) 012(s.0) Omnk (s.a))
1/2
Fi (s.a) ب (s.a)
We learn a neural network: SS tions 8 to tensor corresponding to 3 (8,-) - the action-values of the full set of successor features evaluated under T... This network is illustrated below:
This network maps an environment observa-
1074
State
1004
where both environment state s and policy specifier Pty are mapped into 1024 unit latent representa- tions before being multiplied by each other and then followed by two more 1024 unit layers before a tensor representing i ( (x is output. All internal representations are preceded by leaky-relu activation with hyperparameter 0.1.
Behavior To deal with the fact that our successor feature matrix encodes multiple distinct policies, given by (s)(s) = argmax, Qf, (s;9k)). at the beginning of each environment episode we randomly select a feature / bin pair (i.j) and act according to πı.. When storing experience tuples our replay buffer, we include the index ij of the policy that generated it.
Learning Largely, we train this network in a manner similar to standard DQN (Mnih et al. 2015) along with dueling and double network architectures (Wang et al. (2016) Van Hasselt et al. [006 6) and huber loss for improved stability. We depart from standard architectural choices in how we update each successor feature. For given experience tuple: (s, a, ii) (where encodes the policy that generated the transition), we update the entire successor feature vector, th (s,a) by minimizing:
Th (s,a) - (1)(),)) is (5'
13

---

Under review as conference paper at ICLR 2020
Sample complexity On average agents require around 10 min steps of interactions with the environment to learn the matrix of successor features. This is then followed by around steps per downstream task to solve the regression problem specified in Eq. Bttat is required to apply GPI. This in contrast to > 500 k steps per task required by DQN to reach the same level of performance. Hence, the 10 min steps invested in the ERL stage (m++1)* downstream tasks, where m is the number of feature bins, and is the number of disentangled latent dimensions. Indeed, our GPI framework takes the following number of steps solve all feasible tasks in an environment: (1000+ 5x (m+ 1)") + 1e+44, while the equivalent number for DQN would be (m0 +1e++4 Hence, the data efficiency of our approach becomes apparent the number of downstream tasks is at least 23, since 200 5 + 5 (m + 1)* < 5 (m + 1)* 5 9 (m-+ 1)* and (m + 1)* 22.2.
relatively small cost for the data efficiency gains
14


Reviews:
[{"review_id": "H1x4DQURYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem of policy transfer in reinforcement learning, which is an extremely relevant open problem in RL, and is being actively studied by the community.  The authors propose a framework for discovering a set of policies without external supervision which can then be used to produce reasonable performance on extrinsic tasks.  The work exhibits originality in that it shows that disentangled representations, learned by intrinsic rewards,  can lead to learn behaviours that are transferable to novel situations.  \n\nAlthough the problem talked here is of high relevance and the approach proposed is original and supported by theoretical results, I am leaning to reject for the following reasons:\n\n- Missing connection to some existing works in the literature. In particular, it seems that there is a link to previous works that focus on discovering reward agnostic options (such as [1]).\n- Clarity. The method description is somehow difficult to read, mainly because some variables are introduced without explanation/definition. On page 2, please define n and explain the choices of m and k. The font of the axes and legends in Figure 4 is too small, not readable when printed.\n- Experiments: My first concern is that I do not think I would be able to reproduce the results solely given the paper and supplementary material. It would be necessary to either have access to the code, or a very detailed implementation report.\n- I would have loved to see either another experiment or at least an intuition on how the framework extends to a very different domain. If not, it should be made clearer that this framework works on 2d domains, where the tasks are navigation tasks. (I am specifically referring to the representation learning phase).\n\nMinor comments:\nPage 1, first paragraph “(controlling the position of fruits and nuts)”\npage 5, below (9): shouldn’t it be “While \\pi_w is not …”?\nPage 8, last sentence of Sec 5, “could only learn to perform”.\n\n1. Machado et al, EIGENOPTION DISCOVERY THROUGH THE DEEP SUCCESSOR REPRESENTATION, ICLR 2018 \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByenD63fqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to pre-train policies on some goal-reaching tasks, and then leverage the associated successor features to improve the learning of a new task. The method heavily draws from the Generalized Policy Evaluation/Improvement framework without adding much to it. The only relevant point would be showing (as the title indicates) how to obtain disentangled cumulants, and whether they help transfer to new tasks. Nevertheless, both the definition, the full method, and the claimed benefits are quite ambiguous.\n\nAmong other concerns showing that the theory needs more formal treatment, the pillar definition of “Optimal independent controllability” is very confusing because it seems to depend on “a trajectory generated by following \\pi_i^*”. But what If the environment is stochastic? Then following that policy might give different trajectories! This definition needs to be revisited. More concerning examples are given at the end of this review.\n\nOn the experimental side, Fig. 4 is the only reported result, and it has an x-axis that is not clearly explained. What are the “steps (min)”?\nIt is also not clear what they mean by the “off-diagonal trick”, which seems so important for the good performance.\nFurthermore, it seems that their method doesn’t really learn anything new in most of the tasks, it just stays at the same performance that is started with after the whole pre-training steps. It is not clearly stated how much computation effort is required to obtain the desired cumulants, and this invalidates quite strongly any result they report. Even if there’s no “reward” needed during the pre-training, which arguably is not even true because you do need the rewards related to whether you have achieved a specific change in a feature!\nIn fact, it would be greatly appreciated if the “final” tasks could be expressed in a similar notation than the rest of the pre-training tasks, or vice-versa. As far as I understand, the pre-training tasks consist of making a certain feature fall into a certain subset of its possible values. Can’t the final tasks, like “move the agent to the top right” be also expressed in that form. The link between the two kinds of tasks needs to be much more explicit to be able to assess the relevance of this work.\n\nFinally, they only test their algorithm on Spritworld, which is a small discrete state-action space environment. Even if they try different kinds of tasks in this environment, more detailed analysis or more extensive experiments are needed to assess the benefits of the proposed approach.\nThis is particularly timely because their method relies on a discretization of some given features that represent the state, which will probably not be very practical in higher dimensional environments.\nFinally, I would like a comment on how this method interacts with discrete versus continuous action-state spaces.\n\nMisc comments:\n- Why do the authors introduce the terminology “Endogenous RL”, and then say it’s the same as doing RL with intrinsic motivation? This seems like introducing a new name for the same concept, which seems pointless and confusing.\n- The connection with “latent learning” of Tolman 1984 is very unclear.\n- There’s a “Representation Learning” section, but it’s not clear at all whether any features are actually learned, or whether the features are actually hand-defined. Is the number of features n also hand-defined?\n- There might be a typo in the first sentence after equation (9): “While \\phi_w is not guaranteed to be optimal with respect to \\phi_w”.\n\nBecause of all these concerns, I suggest the paper to be weakly rejected. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xqEuK0Kr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tackles the challenging problem of transfer learning and few shot learning in RL setting and provides some theoretical guarantees for the downstream task coverage. \n\nThe paper structure can be further improved by adding a background subsection on successor representation (SR) in RL; SR is not a very well known representation in RL and a brief subsection on that can help the reader in understanding the motivation behind using it. In terms of related work ,another work which can also be mentioned (although not directly related) is “DARLA: Improving Zero-Shot Transfer in Reinforcement Learning” which also uses disentangled representations for zero-shot transfer learning. The paper also needs to be more clear in terms of contributions; it seems that there is a significant overlap between this work and (Barreto et al., 2017, 2018); some clarification would be helpful here. \n\nIn terms of empirical results; the authors can also compare with other transfer learning methods in deep RL such as Hansen 2019 or Nair 2018 or explain why these are not reasonable baselines. Also the results for DIAYN are a bit surprising to me since in all the experiments the performance of the method is underwhelming; this is especially surprising because in the original DIAYN paper the method performed well in reasonably complex tasks. Can you provide an intuition on why DIAYN performs poorly even in the agent tasks. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The author propose a method to first learn policies for intrinsically generated goal-based tasks, and then leverage the learned representations to improve the learning of a new task in a generalized policy iteration framework.  The reviewers had significant issues about clarity of writing that were largely addressed in the rebuttal.  However, there were also concerns about the magnitude of the contribution (especially if it was added anything significant to the existing literature on GPI, successor features, etc), and the simplicity (and small number of) test domains.  These concerns persisted after the rebuttal and discussion.  Thus, I recommend rejection at this time.
Decision: Reject