Generalizing Successor Features to continuous domains for Multi-task Learning
Authors: Melissa Mozifian
Dieter Fox
David Meger
Fabio Ramos
Animesh Garg
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement learning
multi-task learning
representation learning

Abstract: The deep reinforcement learning (RL) framework has shown great promise to tackle sequential decision-making problems, where the agent learns to behave optimally through interactions with the environment and receiving rewards.
The ability of an RL agent to learn different reward functions concurrently has many benefits, such as the decomposition of task rewards and skill reuse. One obstacle for achieving this, is the amount of data required as well as the capacity of the model for solving multiple tasks. In this paper, we consider the problem of continuous control for various robot manipulation tasks with an explicit representation that promotes skill reuse while learning multiple tasks, related through the reward function. Our approach relies on two key concepts: successor features (SF), a value function representation that decouples the dynamics of the environment from the rewards, and an actor-critic framework that incorporates the learned SF representations.
We propose a practical implementation of successor features in continuous action spaces. We first show how to learn a decomposable representation required by SF. Our proposed methods, is able to learn decoupled state and reward  features representations. We study this approach on a non-trivial continuous control problems with compositional structure built into the reward functions of various tasks.

Under review as a conference paper at ICLR 2022
GENERALIZING SUCCESSOR FEATURES TO CONTINU- OUS DOMAINS FOR MULTI-TASK LEARNING
A Anonymon moonymous authors Paper under double-blind review
ABSTRACT
The deep reinforcement learning (RL) framework has shown great promise tackle sequential decision-making problems, where the agent learns to behave op- timally through interactions with the environment and receiving rewards. The ability of an RL agent to learn different reward functions concurrently has many benefits, such as the decomposition of task rewards and promoting skill reuse. One obstacle for achieving this, is the amount of data required as well as the capacity of the model for solving multiple tasks. In this paper, we consider the problem of continuous control for robot manipulation tasks with explicit representation that promotes skill reuse while learning multiple tasks with similar reward func- tion. Our approach relies two key concepts: successor features (SF), value function representation that decouples the dynamics of the environment from the rewards, and an actor-critic framework that incorporates the learned SF represen- tations. We first show how to learn a decomposable representation required by SE: Our proposed methods, is able to lear decoupled state and reward feature repre- sentations. We empirically study this approach on non-trivial continuous control problems with compositional structure built into the reward functions of the tasks.
INTRODUCTION
Reinforcement learning (RL) tackles sequential decision making problems by defining optimal be- havior through a reward function, where the agent learns how to behave through interacting with the environment and receiving rewards. The ability of RI. algorithms ent, yet related reward functions, has a great potential to realize more data-efficient algorithms with the capability transfer to new reward functions. In this paper we look at one particular type of generalization, where the reward function itself changes, however the underlying dynamics of the environment remain the same. This setup flexible enough to allow transfer happen across tasks, by appropriately defining the rewards which induce different task decompositions. This type of task decomposition potentially allows the agent to tackle more complex problems than, would be possible were the tasks modeled as single task. We are interested in setup where the agent is exposed to multiple tasks i.e. tasks with different reward functions. In a multi-goal setting, differ- ent reward function locations. a multi-task setting, the difference can be intricately designed in the reward function. For instance reward function that determines walking forward vs walking backward. We argue that these differences in the structure of the reward function are difficult to capture within a goal or context-conditioned RL frameworks (Sodhani al. 2021).
generalize across differ-
simply be the difference in the euclidean distance to different target goal
In the context of robotics, generalization across tasks is crucial. Consider an agent playing ball games with racket Figure [ An agent trained dribble the ball hitting the ball, should be able to quickly Iearn to play squash, as many of the skills such as approaching and hitting the ball are shared in more complex task of playing squash. From the learners perspective, all these tasks share the same common properties, the ball falls to the ground due to gravity, depending on heavy it is, and it moves with certain velocity when it is hit by the racket. In other words, all these tasks share common dynamics. What changes is the small details in the reward function. For instance the difference between dribbling a ball vs hitting it against the wall, can be the rotation angle of the racket and the amount of force required.
If was possible to learn a representation that could decouple such discrepancies between the re- ward functions, i.e. decoupling the task dynamics and task-related dynamics, could train an

---

Under review as a conference paper at ICLR 2022
agent that could re-use the learned representation and quickly fine-tune itself the more task- specific representation and achieve a faster learning. Successor features (SF) (Barreto et al. 2017) one framework that enables such decomposability of representation, explicitly built into the RL formulation. The main goal of this framework to promote a desired property where instead of be- ing posed as a decoupled representation learning problem, transfer is instead integrated into the RI. framework as much as possible, preferably in a way that is almost transparent to the agent. SFs in theory, enable fast transfer between tasks that differ only in their reward function. The advantage of using an SF framework over model-based RL where one learns models of the reward function, is the ability of dynamics representation re-use which decoupled from the task ses--speiffic representation.
Our main contribution is to address the gen- eralization and expensive inference problem in the classical SF frameworks coupled with GPI (Barreto et al. 2017). We show that a sim- ple architecture can provide solution to this feature learning problem and demonstrate the effectiveness of method compared (Bar- reto et al., 2020) in more challenging contin- state and action setting. The majority of the existing work using SFs, operates under the
discrete action setting or under the GPI setting optimizing set of policies which in practice are hard to apply on real robotics applications. To the best of our knowledge, our method is the first to show the applicability of SFs cou- pled with appropriate representation learn- ing mechanism solving challenging continuous control tasks. We show that simple modifications to an actor-critic framework be casily coupled with SFs and empirically demonstrate the efficacy of our method.
Figure 1: Agent learning various related skills.
To summarize, contributions are as follows: First we propose practical implementation of SF framework for continuous state and action domains in the context of actor-critic architecture. Secondly propose robust method for learning the representations to learn disentangled representations for the state-space vs task-specific representation of complex nonlinear reward functions. Finally we demonstrate the efficacy our method a ranging from tasks including classical continuous control 2D reacher domain on DM control suite (Tassa et al., 2020) to more challenging 3D reacher and manipulation tasks with the Sawyer arm on benchmark (Yu et al. 2019).
and with the ability
BACKGROUND
2.1 REINFORCEMENT LEARNING
We assume the interaction between agent and environment can be modeled as a Markov Decision Process (MDP (Puterman 1994).. An MDP is defined as tuple M (S.A, p. R, 7) with state space and action space A. For each s E S and a E A the function p(-[s, a) gives the next-state distribution upon taking action in state where p(-1s, a) is referred to the dynamics of the MDP.
The random variable R(s, a, s') determines the reward received the transition s 3 s'. Usually we are interested in the expected value of this variable, which is denoted by r(s, a, s'), and y E [0, 1) weighs the importance of future rewards. The agent's goal is find policy S A, that mapping from states to actions, that maximizes the value of every state-action pairs, defined as
(²( = E Σ r'r(Sit;: Arti S( = A, =
(I)
where S, and A, are random variables indicating the state occupied and the action selected by the agent at time step t and [=].] denotes expectation over the trajectories induced by π. The function Q" (s,a) is referred to the "action-value function" of policy æ.
2

---

Under review as a conference paper at ICLR 2022
RL algorithms based on dynamic programming build two fundamental operations, policy evaluation which is the computation of Q" (s. the value function of policy я on task with reward and policy improvement theorem (Bellman 1957). Once policy has been evaluated, we can compute a greedy policy x' (s)) E arg max, Q (8,a) that is guaranteed to perform at least as well as π, that is: Q" (s, a) 2 Q* for any (s,a) a) E SSS. A. The computation of x' is referred as policy improvement.
2.2 MULTI-TASK RL & TRANSFER LEARNING
In practical situations, agents often face multiple related tasks, such the robot learning numerous skills in Figure T We define tasks M, drawn from the set M. Then, the goal of multi-task learning
to find x optimal policy for each MDP M, with corresponding optimal value function Q Barreto et al. (2017) extended the policy improvement theorem to the scenario where the new policy computed based on the value functions of of policies and referred this as generalized policy improvement (GPI). Suppose the agent has computed n policies corresponding to Q*1 QT ,.... QT action-value functions. Let Qmaz max; QT and define (s) < arg max, Qmaa (s, a) for all E S, then Q2 (s, a) 2 Qmaa a) for all (s, a) E S x A. The only caveat is that it is a waste of computation to compute the value functions of कं. m2; ... nzz This approach becomes appealing we have way to quickly compute the value functions of the policies on the task M
3 ACTOR-CRITIC SUCCESSOR FEATURES
This section will describe Successor Features and their previous use in the discrete action setting. We will then explain our extension, through Universal Value Function Approximators and an Actor- Critic approach, to learn useful Successor Features and corresponding policies for high dimensional multi-task continuous control.
3.1 SUCCESSOR FEATURES DECOMPOSITION
Barreto et al. (2017) proposed simple reward model which leads to the generalization of successor representation (SR) proposed in (Dayan 1993). The key assumption is that the reward function can approximately represented as linear combination of learned features ((s). The successor representation (SR) (Dayan) 1993) is a representation that generalizes between states using sim- ilarity between their successors, that is, the states that follow the current state given the agent's policy. The generalization of SR with function approximation is referred to Successor Features (SF) (Barreto et al. 2017 of (s, a) under policy т. Following (Barreto et al. 2017 2018 2020), let S A I be arbitrary function whose output will see as "features" We assume that there exist features such that the reward function can be written as
r(s, 0, 5') = ((8, a, a,,')²
where (bs, a, s') E Rdd are features of (s, s') and w E Rd are weights. Intuitively we can think of d(s, a, s') as salient events that may be desirable or undesirable to the agent. Based on Eq we can define environment M (S,A, as
M 0 = ASp.S,
o(s, 8'""
(3)
that is. M is the set of MDPs induced by through all possible instantiations of w. SFs make it possible to compute the value of policy π on any task M, E M by simply plugging in the representation vector w, defining the task. Specifically, if we substitute Eq in the definition of action-value function of a policy we have
(*, = E²
rtt + 7T42 + ...|SI = A =
ETT
df w + 2442 :...|[:= =3 =
T
E
1 = A, = w
(T, aT

---

Under review as a conference paper at ICLR 2022
One benefit of doing so is that if we replace w; with w in Eq we immediately obtain the evaluation of т on task M,. This way only relevant module must be relearned, when either the dynamics or reward changes. The key insight of SFs is that linearity of rewards Iw with respect to the features which gives us the decomposition of the action value of policy и on task rw- In the GPI setting when
the agent is presented with new task Mn+1, it needs compute 10 Q 1: Q741): J. that is, the evaluation of each क. under the new reward function induced by W This in turn would require applying the GPI theorem to the newly-computed
of value functions that will give rise to a policy that performs at least as well as the policy based on any subset of these. Hence (Baretto al. 2017) proposed to incorporate SFs where the reward function changes rn+1 (s, a,s')
(18, G, (n,s') s')' Wn+1, as long as we have the correct Wn+1 we can compute the value function of ar by
simply computing (s, a) = 3 (s, a)² Wn+1. This reduces the computation of all Q to the simpler supervised problem of approximating Wn+1 (Barreto et al. 2020).
Although at first glance, GPI and SFs seem tangled, this characterization of SF does not depend on GPI framework itself. Thus, SF can be used in any RL framework where such decomposition of reward is viable. Furthermore the combination of SFs and GPI provides an elegant framework for transfer in multi-task setting. However the computational cost of optimizing policies per task is prohibitive in more complex multi-task setting where in the size of task set M is large.
3.2 UNIVERSAL SUCCESSOR FEATURES AND TASK INFERENCE
We start our discussion by assuming the existence of appropriate representations o and that are a decomposition of the reward function. We then turn our attention to the policy learning phase that will make use of these representations generalise to unseen tasks, the agent needs to be able to identify and exploit some common structure underlying the task. Two possible sources of structure in this scenario are: i) similarity in the space of tasks, i.e. reward functions, and ii) the shared dynamics of the environment. One framework that enables exploiting structures are universal value function approximators (UVFAs) (Schaul et al. 2015). UVFAs extend the notion of value functions to also include the description of a task, thus directly exploiting the common structure in the associated optimal value functions. As discussed in (Borsa et al., 2018), UVFAs and SF & GPI address the transfer problem in quite different ways. With UVFAs, one trains an approximator ((s, w) by solving the training tasks w E MM using any RL algorithm of choice. [Borsa et al. (2018) proposed universal successor features approximators (USFAs) with GPI. They show that by combining USFAs and GPI can outperform SF&GPI on grid-world navigation problem. Since majority of existing work explores SFs the context of GPT or Q-learning under discrete action setting. in this paper we will examine a more practical UVFA style framework targeted at solving more challenging continuous action control problems.
substitute for the reward signal. For RL agent to
In this work, we ask the following questions: Can we incorporate SFs and train policy which can yield better generalization across multiple tasks? Do SFs introduce implicit composability at the policy level? Composability of controllers is specially important for real-world applications, where reuse of past experience can greatly improve sample efficiency for tasks that can naturally be decomposed into simpler sub-problems. For instance, policy for pick-and-place task can be decomposed into (1) reaching specific target, (2) grabbing an object, (3) avoiding certain obstacles. Such decomposable policies can be learned offline without the need to interact with the environment. Although we are not using an explicit notion of compositionality as in (Haarnoja et al. 2018a), we count on such properties emerging from the representation itself.
3.2.1 ALGORITHM DERIVATION
In this work we will be extending Soft Actor-Critic (Haarnoja et al. 2018b). by two means of generalisation. That by combining. UVFA and USFA to learn to generalize values over multiple tasks. The structure of the RL problem creates number of challenges. Without having access to the target function, we generally use the approximation itself to build targets and many of the techniques like replay buffer and target networks and optimizing two target networks by selecting the minimum which leads to lower variance, are just strategies to remedy this instability (Mnih) et al. 20155 Fujimoto al. 2018). Unfortunately since we are dealing with target functions, these instabilities are exacerbated. We elude this problem by taking the average of s along the dimension of the features and the minimum is determined based on this average value.
4

---

Under review as a conference paper at ICLR 2022
This causes further instability during the policy learning, and longer delayed critic target updates were required. Similar to Hunt et al. (2018), we define action retion-dependent SF to include the entropy of the policy
s Σ ' a
3 (st, - de + Ex
(5)
The max-entropy action-value of π for any convex combination of rewards w then given by (5, a) = 2 (s, a)w. For more details see Appendix A.1
3.2.2 LEARNING SUCCESSOR FEATURES IN CONTINUOUS CONTROI.
We describe how build decomposed representation required for & and w. To make dis- cussion more concrete, consider a simple 2D goal reaching environment. We formulate the problem in Eq. 2 as computing an approximate as multi-task problem, solving the following approxima- tion:
Mss a, 8'" (s(,, a, for = 1.2..... D
(6)
We can start our assumption by considering the scenario where such representations Ø E R" and w exists that satisfy Eq. [ exactly. For instance, can define d; as an indicator function signaling important event in the state features e.g. whether an object of type has been picked up by the agent (Barreto et al. 2020). Analogously, in a continuous control setting, we can consider goal reaching non-linear
task with a non- linear reward function. A common reward function for simple reach task can be defined as the euclidean distance between two vectors, that is, the position of the agent's s link and the goal location. Given the state vector μ (xr, v) and goal vector g = (x, y), we define the following reward function,
r(u. g) = 111 111p-10 g|l =1/11 + 2g" 11a11
It is trivial to see the corresponding ((u) and w(g) can be recovered as follows,
1 ((g)
- |و 1 2g -1
Φ(n)= = 11 1|
=
(8)
Thus given such decomposition, we can recover the reward function as the linear combination of w(g). w(g). Such representation, for this particular reacher task, immediately gives solution to compute gोt And by changing the target goal location associated with each reacher task, one can create different tasks, where the only part of the representation i.e. goal location, changing is w, relying on the assumption that the representation captures any task-related features.
-00
-22
-100
SAC
ACSFFUPNFFW
50
100
150
200
250
episcce
Figure 2: 1-Goal Reacher Task. Comparison between learning Successor Feature based SAC algorithm with pre-defined & and regressing either & and These results sanity checks that given appropriate representations & and w, it is possible incorporate SF within our Actor-Critic continuous action framework.
5

---

Under review as a conference paper at ICLR 2022
Now that we have defined 6, we turn to the question of how to determine appropriate policy и to solve the task. We will start with what is perhaps the simplest case, for a single-goal reacher task. In our experiments, Figure _ we compare the performance of SF integrated with SAC. As a sanity check, we alternate between keeping either & or w fixed and regressing the other. This experiment ensures that our objective is able to learn the decomposed representation for this reward function for each component individually. These results confirm that, given the appropriate successor features, our policy learning mechanism should be able to solve the task, and perform as close to the optimal policy learned via the original reward signal. However most RL tasks have complex reward functions and it is not feasible to always aand-engieeer these features. Instead, we are interested in learning these as a pre-training stage.
Barreto et al. (220)) proposed an approach to learn such representation. Given a task v, we are looking for w E Rd that leads to good performance of the generalized policy Te (s: Suppose we have a set of m sample transitions from a given task, {(,,, as ry s)) Then based on Eq
we can infer w by solving the following minimization:
min Σ l0(s;: az " r/1". :=1
99
where p 1 and we may want to consider the inclusion of regularization term in this objective function. If we handcraft o. we can obtain a solution úú which can be plugged in during the rep- resentation learning phase. It turns out that Eq 9 can be generalized also allow é to be inferred from data. Given sample transitions from tasks, {syy, aey, Tig. s1 Ha) with j = 1,2, ...,. k. we
can formulate the problem as the search for a function @ : S A S R and is vectors w, E R satisfying,
min
(B1)) Bijs slyy
r'y
(10)
where 2 1. Note that the features Ø can be arbitrary nonlinear functions of their inputs. As discussed in (Barreto et al. 2020) this objective can be decomposed into single objective for learning á and w stage-wise where the initial learned representation á can be re-plugged in to learn w. More generally, the problem in Eq TG be solved as a multi-task regression (Caruana 1997).
Barreto et al. 2020) proposed the following procedure for learning & and w. First, a random policy is used to collect data from tasks WT, W2...., Wg. They propose to use neural network with the number of output units or "heads", corresponding to the number of tasks. The top layer of this joint network represents o, and hence each output unit represents r, o"T w,, Given sample transition (s, a, Fi, s'), only the i-th output unit of the network has access to the branch where the sample is used modify w; and ςb. Once this optimization is done, the weights w; are discarded and the same network is used re-compute w, while keeping the learned representation ø from previous iteration frozen. This process repeats the optimization process solving for equation 9 but this time solving for w. When learning w, "fresh" output units are added to the network i.e. the head branches of the network weights are randomly initialized again and using gradient descent the associated weights w are modified, while keeping & fixed. Having computed Φ and w, they resume to learning the policy in their GPT setting. (Barreto et al. 2020) demonstrated the effectiveness of this procedure using simple tabular grid world navigation task. this paper, we show that this procedure does not allow for learning more complex representation such as a simple reacher task with euclidean distance reward function. Instead, we propose the following procedure which deemed more appropriate for more complex control tasks with continuous state and action spaces.
3.2.3 JOINT TASK INFERENCE
Following (Barreto et al. 2020), we propose optimizing an approximation to the objective in Eq T0
where the representations & and w are learned jointly, as opposed to the two-stage optimization procedure proposed in (Barreto et al. 2020). Given set of transitions {(s;, a4, T/ $ )mm ideally covering the space of state space for all the tasks seen at training time, we use two neural networks de and wa, where the function ø takes as input the tuple (s, s') and the function w is a function of the task itself, in a multi-goal setting, this can be the position of the target goal or task ids. Given these two networks, using stochastic gradient descent we can approximate Eq by optimizing,

---

Under review as a conference paper at ICLR 2022
W
W
Task
Task Atgoo
W
(a) Representation learning
Barreto
2020
(b) Our representation Learning Module.
Figure 3: Architecture component of cach representation modulc.
172
min
min 129
(s(s azj. ((() rtyl
(11)
where á and w are optimized jointly by considering their output as d² w ss This decomposition has two advantages over (Barreto et al., 2020). Firstly, it is easier to decompose the environment dy- namics information fed to the network via Φ(s, a, s') and task-specific information g via w(g) where each these components are fed the relevant information and the composition these corresponds the reward approximation. This is not possible with the architecture proposed in (Barreto al., 2020) since both é and w share the same input and the decomposition of these representations are made explicit during the stage-wise optimization procedure. In addition, single-stage optimization makes the inference task simpler to train and evaluate. We will illustrate the results of both these approaches in our setup for simple single-goal and multi-goal reacher task. Figure shows the architecture of the two methods.
4 EXPERIMENTS
Figure 4: Continuous control tasks we are considering in this work, starting with simple link reacher task to more complex metaworld reacher and door close tasks.
Average Return
-40
Averag
-30
SAC
SAC ACSF LoFFHHoMHHO
MCSFJohIHHo0ML ASSE
(a) Policy evaluation reacher environments.
(b) Policy evaluation reacher environments.
Figure 5: Multi-Goal reacher task regression & and w comparison.
Our experiments consist of learning continuous control robot manipulation tasks shown in Figure In the case 2D link reacher, we fix the goals, separated into a set training goal targets and test goal targets, and for metaworld, consider single goal tasks where the tasks are different.

---

Under review as a conference paper at ICLR 2022
When learning decomposed representation of the reward function, the function itself should ideally be a dense function. Otherwise it hard to learn a good representation with a sparse rewards as also pointed out by (Machado et al. 2020)). For this reason, for the default 2 Reacher task in DMSuite to a simple 1.2 norm squared reward function which easier learn than semi-sparse reward function. The original reward function of this task is sparse, where dense reward are given when the agent is in the proximity of the target goal.
modify the original reward function
For each task, first, we need to learn representations and w. Barreto et al. (2020) proposed to use random policy to collect the initial data which could be sufficient for simple grid world navigation task. In our setting, we found that simple random policy does not allow full coverage of state space. is possible that a random policy a small grid-world setting enough however we found this be insufficient in a continuous control setting, allowing learn generalizable é and w. Therefore we used a mixture of expert data and data generated by a random policy train & and
Figure shows comparison between the proposed method in (Barreto et al., 2020) and our method where we compare SAC policy trained with the original reward, our method with ACSF where and w are learned jointly using the training scheme, referred to as ACSFJJonnHeauMMLP proposed (Barreto et al., 2020 vs our method for learning the joint representation. The task for this experi- ment multi-goal reacher where for the architecture in (Barreto et al. 2020), each goal í.e. task is represented by a separate head of the network. We can see that training a joint architecture becomes harder for two reasons. Since the joint network takes in the input (s, a, s' g), it is not possible to disentangle the inputs to the module learning and the module w. As expected, the performance drops even further, disentangle these observations for the dynamics learning vs any task-related part of the observation. In addition, the training methodology in (Barreto tation learning component, meanwhile one must ensure overfitting is not happening at each stage of learning, and this hard evaluate in general without having access any prior ground- truth.. Although this is not a direct comparison of the GPI method in (Barreto et al. (2020), the main ob- jective of these experiment is to evaluate the representation learning procedures embedded into our policy learning framework.
remove the goal information from the observations tuple. We prefer to
2020) is 3-stage training for the represen-
FigurerJssowws the generalization capability our method evaluated on seen goal locations train time vs unseen goal locations. Finally Figure7]showe the performance of our method compared to goal-conditioned SAC policy. Figure 7a shows the performance of both methods on the reacher task where the goal location is perturbed but this is still single goal task. Figure 7b shows the perfor- mance on door close task and finally Figure [7 [c] shows when these tasks are learned simultaneously. These results confirm that method works in par with goal-conditioned policy and it has learned the correct features which in turn enable learning robust policies.
-20
-20
Average Relum
-30
-40
Average
-50
-60
-60
-70
-80
ACSF
ACSF
100
200
300
400
600
100
200
300
400
500
episode
episode
(a) Policy evaluation on seen goal targets.
(b) Policy evaluation on unseen goal targets.
Figure 6: Multi-Goal Reacher Task comparison between SAC and ACSE, This plot shows the training for the train and evaluation setup for the 2Dlink reacher task. The training and evaluation on same target goals seen at train time show similar performance between a SAC policy and a ACSF policy. However we are interested in the generalization result in Figure ob where ACSF outperforms SAC on unseen target goals.
8

---

Under review as a conference paper at ICLR 2022
5000
68300
4230
8000
40:0
Raamm
50.00
Avarge Reamm
3000
2000
2200
1000
SACThecter
SacOberChee
1030
200
400
1980
200
601
1939
X00
300
480
700 801
(a) Reacher task.
(b) Door Close task.
(c) Reacher & Door Close tasks.
Figure 7: Metaworld reacher task comparison SAC and ACSF on Reacher and Door close tasks individually, and trained jointly. These results show our method performs par with learning without an explicit SF representation.
5 RELATED WORK
Multi-task RL and representation learning in RL are important topics that have generated large body of literature and as (Dabney et al., al.,,22211 (02T] formally points out, there remains challenging aspect of RL when learning representations. There exist various techniques that inject tasks information directly into the definition of the value function. UVFAs have been used for zero-shot generaliza- tion (Mankowitz et al., 2018) and learning set of fictitious goals previously encountered by the agent (Andrychowicz et al. 2017). HHarnoja et al. (2018a) proposed learning with multiple ob- jectives where they define compound task expressed in terms of the sum of individual rewards. More recently, (Sodhani et al. 2021) proposed an approach for a contextual multi-task RL as a way to incorporate task metadata, contextual information. One other common and successful way to approach the representation learning problem is through the use of auxiliary tasks: additional pre- diction problems that shape the representation used by the agent (Jaderberg al. 2017 Bellemare et al. 2019).
Another body work relates the use of SFs. Kulkarni et al. (2016) presented a generalization of successor representation in a deep reinforcement learning framework, based on DQN with discrete action space. Similar in the spirit, (Zhang al. 2017). ). proposes deep RI. architecture built on DQN to solve robot navigation task, Lehnert & Littman (019) draws connections between model- based RL and successor features by analyzing properties different learned latent state spaces. More related to our work is Ma et al. (2020) which combines goal-conditioning with SFs, where SFs are learned end-to-end using temporal difference learning methods. However in their setup, learning these successor features is done implicitly inside their architecture. In addition. they the true rewards during policy learning and the additional auxiliary loss for learning e demonstrates additional stability over DDPG baselines.
Barreto et al. (2019) showed how to combine SF&GPI with deep learning using the reward func- tions themselves features for future tasks. Barreto al. (2020) introduced the representation learning component of SF's as opposed to using hand-crafted features as done in prior work. Hansen al. (2019) introduces Variational Intrinstic Successor FeatuRes (VISR) which shows that behav- ioral mutual information (BMI) maximization provides a solution for learning the successor feature representation. They evaluate their method as an unsupervised pre-training technique. Hunt et al. (2088) introduced SF in the context of maximum entropy framework, extending GPI theorem to max-ent objective. Blier et al. (202T) formally derive a temporal difference algorithm for succes- sor state and goal-dependant value function learning with function approximation. [Zahavy et al. (2021) considers specific class of policy composition called set improving policies (SIPs). Gimel- farb et al. (202 l) proposes risk-aware successor features (RaSF) integrated into generalized policy improvement framework to maximize entropic utilities. Touati
Ollivier (202I) introduces the forward-backward backward (FB) representation of the dynamics of an MDP, learning two representations in- stead one in the SF framework. Their setting is also more tailored towards a goal-oriented RL problems with discrete action space.

---

Under review as a conference paper at ICLR 2022
CONCLUSION
Our results suggest that our method can successfully incorporate SFs representation for learning continuous control policies. Learning decoupled representations has been a longstanding challenge in RL, and this work must continue in continuous control tasks, especially for the multi-task setting with continuous goals. Our work sheds light on possible ways of incorporating SFs and their capa- bility to solve more challenging RL control problems. Unlike the framework proposed in (Barreto al., 2020), which explicitly tailored for transfer, this is not yet as straightforward critic architecture. Future work in this area includes more stable online feature learning and scaling more complex reward functions. By demonstrating that ACSF performs well large continuous tasks, believe our approach is an important step in the direction of composable representations.
our actor-
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abheel, and Wojeiech Zaremba. Hindsight experience play. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL /tLpttts//prroeeeiingg neurips.........p..p............ httpp//ccccpppee////11122 www.aaabbaala3affoooooooooooooo
Paper. pdf
Andre Barreto, Will Dabney, Remi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Has- selt, and David Silver. Successor features for transfer in reinforcement learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: http:///oooeeeeeseeeeeeeee www.eddiiiinnnnneineeeeeeeeeee www.nnuriiss.. httpp//awwg.....r.o.....pppppppa cctppappee/111//iieeeee..
9b8c05 Paper.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 501-510. PMLR, 10-15 Jul 2018. URL
lttpp://proceedingg www.................mm.m. prwss/rsseeee....
André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences, 117 000:300799-0087) 2020.
André Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zídek, and Rémi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement, 2019.
Mare G. Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Taiga, Pablo Samuel Castro, Nico- las Le Roux, Dale Schuurmans, Tor Lattimore, and Clare Lyle. A geometric perspective on optimal representations for reinforcement learning. In NeurlPS, pp. 4360 1660-4371,220 2019.
Richard Bellman. Dynamic Programming. Dover Publications, 1957. ISBN 9780486428093.
Léonard Blier, Corentin Tallec, and Yann Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint. CoRR, abs/2101.07123,, 2021. URL https://arxxv
/rg/ abs//101................ab...b.. 07123
Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado van Hasselt, David Silver, and Tom Schaul. Universal successor features approximators, 2018.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Mare G. Bellemare, and David Silver. The malee-impoovement path: Towards better representations for reinforcement learning. In AAAI, pp. 7160-7168. AAAI Press, 2021.
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Comput., 5(4):613 July 1993. ISSN 0899-7667. doi: 10.1162/neo..1993, 1993.5.4. 613. URI.
w2/nneco 1993.999..466
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference Machine Learning, pp. 1582-1591, 2018.
Michael Gimelfarb, André Barreto, Scott Sanner, and Chi-Guhn Lee. Risk-aware transfer in rein-
forcement learning using successor features. 2021.
Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine.
Composable deep reinforcement learning for robotic manipulation. In 2018 IEEE International Conference Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, pp. 6244 6251. IEEE, 20.8a. doi: 10.1109//CRA.2111 8460756. htRhhtpps: /oii. org/10. 1109/ 009//CRAA 201 18 6460756

---

Under review as a conference paper at ICLR 2022
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policc maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and An- dreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018b. htt.htt/p///poceeeeeee
Steven Hansen, Will Dabney, André Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. CoRR, abs/1906.05030) 2019. http:/
org/ abs/1906. 05030
Jonathan Hunt, André Barreto, Timothy P. Lillicrap, and Nicolas Heess. Entropic policy compo- sition with generalized policy improvement and divergence correction. CoRR, abs/1812.00226. 2018. URL URLL:ttp://are wwtp./orggab////aeeee 1812. C2216
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In ICLR. OpenReview.net, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Tejas D. Kulkarni, Ardavan Sacedi, Simanta Gautam, and Samuel J. Gershman, Deep successor reinforcement learning. ArXiv, 2e21166666096666 2016.
Lucas Lehnert and Michael L. Littman. Successor features support model-based and model-free re- inforcement learning. CoRR, abs/1901 11437, 2019. UtI.http:///ariii..o////////////
11437
Chen Ma, Dylan R. Ashley, Junfeng Wen, and Yoshua Bengio. Universal successor features for transfer reinforcement learning. CoRR, abs/2001.04025, 2020. URL httphttts///aiiv.rrr
abs/2001 04025
Marlos C. Machado, Mare G. Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In The Thirty- Fourth-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020,
New York, NY, USA, February 7-12, 2020, pp. 5125-5133, AAAI Press, 2020. URL https:
7/ 7/aaai.ora jjs/ /www/xinxxxe/eeee/eee php/ AAAI www.a//zticl////./ wwt.e//eevvwweeoooeee 5955
Daniel J Mankowitz, Augustin Zidek, André Barreto, Dan Horgan, Matteo Hessel, John Quan, Junhyuk Oh, Hado van Hasselt, David Silver, and Tom Schaul. Unicorn: Continual learning with universal, off-policy agent. arXiv preprint arXiv: 1802.08904 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529-555 533, February 2015. ISSN 00280836. URL. 4236
doi. 0ttt::://aa.do...rr//000000000
Martin L. Puterman. Markov Decision Processes. Wiley, 1994. ISBN 978-0471727828. URL decision
http::/booke. http://books..oggeeeee wwwpggee..o//ooooooooooooooooooo aata// htt.a/arooooeeiiiiono.oooo processes.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accel- erate training of deep neural networks. In D. Lee, Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran As- sociates, Inc., 2016. URL
neurips. htt.eu/rpp....eppp///////////L/. pdf
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1312-1320,
12

---

Under review as a conference paper at ICLR 2022
Lille, France, 07-09 Jul 2015. PMLR. URL http://ppoocednnn............s. Schaul15.herr
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th Inter- national Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research. pp. 9767-9779, PMLR, 18-24 Jul 2021. URL bttps::/ppoceecing....
pzess/v139/sochan122
html
Yuval Tassa, Saran Tunyasuvunakool. Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dm control::: Software and tasks for continuous control, 2020.
Ahmed Touati and Y. Ollivier. Learning representation optimize all rewards. ArXiv, abs/2103.07945 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcemeet learning. In Conference on Robot Learning (CoRL), 2019, URL /ttp:s//aaxll..orr///////111
arxiv. wwtp:/ivwrgg/bb////////
10897
Tom Zahavy, Andre Barreto, Daniel Mankowitz, Shaobo Hou, Brendan D'Donoghue, lurii Ke- maev, and Satinder Baveja Singh. Discovering set of policies for the worst case reward, 2021.
Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep rein- forcement learning with successor features for navigation across similar environments, 2017.
A
APPENDIX
A.1 IMPLEMENTATION DETAILS
The loss function for learning & is:
1
27
LACSF()) = (s S
viss - ulo,s',
(12)
2
where the target is given by
v(O. 8', d) 6(s, a, s') + (
log No ('d(ss))) o' ~ mo(ls')
(13)
where d corresponds to the termination criterion. The gradient with respect to the parameter is
VOLACSF(U) = (ss,,,s, (((ssrr())=(ssar (
- u(s, -v(s,a))((s, s')Vou(s 5') 3)
(14)
which similar to the gradient used (Haarnoja et al. 2018b), with the distinction that 1 is a multi-dimensional matrix rather than vector as the original definition of Q. The dimensionality of db is hyperparameter that needs to be tuned depending on the task observations complexity.
Figure shows the comparison of the setup for our representation learning module compared to [Barreto et al. 2020). First stage requires learning the representations & and w. Ideally, the data used train these modules needs to be diverse enough, covering the state space adequately. We found that learning these representation online tangled with policy learning does not perform well since the representation needs to be able to handle regions where policy behavior optimal. Therefore we resort to using expert policy data. However we still do require the diversity. More intuitively, this is because at the beginning the agents learning experience, a random policy is exploring, is important to have good coverage of the state space that can ultimately lead to the highest performing regions. Therefore we use mixture of expert to random policy data a 75% - 85% of expert and the remaining data generated from random policy with some action noise. We found this ratio to be important for training robust representations that will later on perform well during the SF policy learning stage.
13

---

Under review as a conference paper at ICLR 2022
Table 1: ACSF Hyperparameters
Parameter
Value
optimizer
Adam
Kingma & Ba 2015
learning rate
1.10
discount (7)
0.99
dimension of hidden layers dimension of hidden layers
1024
64
dimension of hidden layers policy
1024
dimension of samples per minibatch nonlinearity
256
ReLU
Actor update rate ? update rate Dimensionality of tb
1
4
4-28 (Task dependent)
We also regularize the w using the layer weight normalization (Salimans & Kingma (016) which simple reparameterization of the weight vectors in neural network that decouples the mag- nitude of a weight tensor from its direction. For the policy learning, we use an open-source SAC implementation extending this implementation ACSF policy.
A.2 APPROXIMATING THE REWARD FUNCTION
We demonstrated our method on tasks with more reaching-style reward functions which are still learnable with our objective function. However this simple objective fails to capture more difficult tasks or tasks with more complex state representations. For instance, method fails to learn good representations for some of the tasks in Meta-World such as picking and placing objects or tasks interacting with object such pressing button. We hypothesize that more complex reward functions are too difficult for the current method to automatically decompose.
A.3 FEW-SHOT TRANSFER
Consider the 3 reaching scenarios presented Figure our experiments we found that, an agent that had been trained on the reacher and door close task, struggled to zero-shot transfer to the third reacher task with an obstacle on the way. The performance had higher variance than we expected and this happened with both SAC and ACSF which could entail an underlying issue with the policy learning. Note that we would expect the agent to adapt to the third scenario through fine- tuning. However in experiments slower learning and, in some cases, reaching sub-optimal behavior due to instability. We found that it is better to train agent the tasks simultaneously
found that the continual learning the third behavior led to
agent trained per task separately. We hypothesise that an agent starting from scratch with random exploration has a better chance at discovering all the skills compared to an agent who has learned reach the first two scenarios. Transferring to the third case, it needs to unlearn the previous behavior and adapt.
Figure Agent learning various forms reaching skills.
hhtps://itthh..cmmmdmiiiaaaaaaap sac
14

---

Under review as a conference paper at ICLR 2022
Algorithm ACSE- Plugging in learned e and w
Input: initial policy parameters
V/ parameters corresponding to th1 and r/2 Replay buffer 'D
Set target parameters equal main parameters
1 f 1/,, 12er.2 + 1/2
for Time to do Observe state s and select action TT(.ss)
Step environment, o ~ p(.1)).0))
D < D U (8t, dt, tr, 94 if it's time to update then do
for in range(numUpdaterr
Randomly sample a batch of transitions, = {(s, a, o, s', d,g)} from
Compute targets for the D-dimensional V functions:
P Star (s,
V plaar
D
D
(s,
Pear
d=1
v(o,s',d) ((s,a,)') +1(11 d)( (min v Ve, (s, To ,9)),
1=1,2
a' ~ (-lss'gg)
Update V-functions by one step of gradient descent using
V vi B
Σ
E B(min
min (s āo(s),)) a log (s)
1=1
Compute the Q -unncttion
Q1 - V tan, (8, 1, g) Tw
Q2 v 2 a, 9)² Tw w
Update policy by one step of gradient ascent using
Σ (min (ss, ag(s), g) alogro (
Ve
B
11122
(s.i)
where (s(s) is sample from Ta(ss) which is differentiable w.r.t. 0 via the reparameteri- zation trick. Update target networks with
Vtarg,i + Pitaagg.s + (1 p)yb
15


Reviews:
[{"review_id": "cHioDbwnmG2", "reviewer": "Reviewer_f8Lk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are a number of concerns about this work.\n\n1. The paper states that it introduces an actor-critic approach to SF in continuous action spaces. The paper outlines the approach to learning the action-value (and successor features which can be used to compute the action-value function) [the critic], but the only details on how the actor (policy) is represented, trained or generalizes is provided implicitly in the appendix. In particular, a key challenge in continuous action spaces is that it is, in general, non-trivial to find the optimal policy even given the action-value function. How to benefit from the SF representation given this challenge is non-obvious (since SF approach provides an action-value function for a new task, but unlike in small discrete action spaces, this does not automatically allow you to infer a good policy). The challenges of doing this in continuous action spaces is outlined in e.g.\\ Hunt et al. (2018).\n\nThe approach used here appears to be (from the appendix) that a goal condition policy is learned. Therefore, after training successor features are not used (since the policy does not directly use them), and the test performance is simply based on goal conditioned policies. This seems like a significant limitation of this approach, since it means that the structure of successor feature representation is only able to be used during training. It also seems a bit misleading to suggest in the introduction some sort of solution to successor features is provided for continuous action spaces, when it is more just ignored and SFs are not used after training (and therefore the claims of more structured generalization don't apply). Given this limitation some discussion about why SAC does not generalize as well to unseen tasks would also be helpful since it is also learning a goal conditioned policy and its not obvious why it should be inferior.\n\n2. I have some concerns about the novelty of this work. Hunt et al. (2018) applied successor features to the max-ent RL framework. Kulkarni et al. (2016) learned jointly $\\phi$ and $w$. Goal conditioned policies are an old idea. I think combining existing ideas can be of interest, if for example, it demonstrates compelling empirical results. However, the empirical results here are not particularly challenging tasks, and there is no comparison with other goal-condition policy learning approaches (such as Hindsight Experience Replay).\n\nIt would be helpful if the authors consider releasing code for reproducing the experiments in this paper. Particularly, as there are a number of implementation details that are unclear.\n\nThis work does appropriately cite and explain prior work in this topic.\n\nMinor:\n\nInconsistent vector notation ($\\mathbf{w}$ is bolded, by e.g. $\\mu = (x, y)$ is not.", "summary_of_the_paper": "This work looks at the use of successor features for solving simple continuous control tasks (in particular, reaching to different locations and door closing). The two contributions they enumerate are a ``practical implementation of SF framework for continuous state and action domains'' and jointly learning $\\phi$ and $w$ (that is the successor features and the task weights $w$). They show their approach outperforms a goal-conditioned SAC baseline on these control tasks including generalizing to target locations that were not used during training.\n", "summary_of_the_review": "This paper puts together an approach to apply SFs to some robotic control tasks. However, the approaches are not particularly novel, the empirical results are not compelling and key details regarding training of the actor are not included.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PdXyrK_EkOp", "reviewer": "Reviewer_Nsua", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**: The authors proposed a principled method for incorporating successor feature into an actor-critic algorithm and the decomposition of successor features as $\\phi(s,a,s)^Tw(g)$ allows it to adapt to new goals as shown through reacher envs.\n\n**Concerns**: \n(i) The proposed decomposition $\\phi(s,a,s)^Tw(g)$ feels very similar to that used in (Ma et. al, 2020). Moreover, (Ma et. al, 2020) also incorporates the learned successor features into an actor critic framework and uses it for continuous control tasks. Hence, I am not sure what novelty the proposed work is bringing.\n\n(ii) The environments used in the experiments are limited (only reacher and door close tasks are considered). Moreover, the performance of the proposed method is very similar to the goal conditioned SAC baseline on meta world tasks (Figure 7)\n\n(iii) At an intuitive level, if $\\phi(s,a,s')$ is capturing the dynamics and $w(g)$ is capturing the reward information, shouldn't $w$ be also conditioned on $s$ as well (i.e. the final decomposition being $\\phi(s,a,s)^Tw(s,g)$). This is because $g$ alone doesn't give reward information. Both $s$ and $g$ are needed for reward information.\n\n**References**\n1. Universal Successor Features for Transfer Reinforcement Learning. Ma et. al, 2020. https://arxiv.org/pdf/2001.04025.pdf", "summary_of_the_paper": "The paper proposes (i) a decomposition for successor features $\\phi(s,a,s)^Tw(g)$ which allows the Q-value function to adapt to new goals easily and (ii) then incorporate the learned successor feature representation into an actor critic algorithm (SAC) for goal conditioned tasks (or for solving multiple tasks). The proposed method is tested on continuous control tasks.", "summary_of_the_review": "Weighing the strengths and concerns as listed above, I am currently recommending the paper for a rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "DC-Jh_Nb2ag", "reviewer": "Reviewer_QCxi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The use of SFs is well motivated in the multi-task learning scenario for continuous domains since they can promote skill reuse and composability. More than that, the proposed method also allows transfer learning among tasks through the task-specific module.\n\n- The experimental evaluation in Figure 6 shows a clear improvement from the proposed method (ACSF) compared to the goal-conditioned SAC policy, which suggests that ACSF is learning meaningful representations that enable generalization across tasks.\n\n- It is interesting the way that ACSF modifies the SAC framework to enable successor features. It requires some detailed design choices to consider double Q-functions and target networks and keep their stability effect.\n\n- The introduced training method (single-stage procedure) is also valuable due to the simplification from the prior method [1].\n\nMajor Concerns:\n\nThere are two general points of criticism that should be addressed during the rebuttal period. First, the paper proposes a method that is quite similar to prior literature (e.g.,  Ma et al. [2]), which is not contrasted or discussed in the paper. Secondly, the experimental section does not communicate the intentions well: given the motivations for this method, the experiments miss the critical gains of it compared to prior methods. Furthermore, it lacks a solid analysis of the results and limitations. In more details:\n\n- The contribution of this paper is fairly incremental, and the novelty is questionable. Indeed, Ma et al. [2] proposed “Universal Successor Features”  to capture the underlying dynamics of the environment while allowing generalization to unseen goals. It also extends to the continuous scenario, which seems to refute the claim that this paper “is the first to show the applicability of SFs for continuous control tasks”, as mentioned in the Introduction. Furthermore, the paper does not analyze [2] in the Related Work section, despite the similarity. Could the authors discuss this paper from the perspective of the proposed method during the rebuttal session, please?\n\n- Although I think interesting the level of details to enable the SAC framework for successor features, these minor decisions are quite straightforward (e.g., applying average to reduce dimensionality, and replicating the Q-function inference for the double-Q scenario). The inference scheme and training procedure are also inherited from the SAC framework. I understand that they will work simpler and better due to other components of the proposed method, but they do not present novel ideas themselves.\n\n- The evaluation methodology described in Figure 5 trains the compared methods in different MDPs: the SAC Policy using sparse rewards and ACSF using dense rewards. In my perspective, the sparsity should also harm SAC generalization, and, therefore, this comparison seems unfair. I would like to ask the authors if they think that this is an issue and why.\n\n- As described in the Experiments section, the proposed method relies on expert data to work. This is a critical variable to consider (especially in terms of the method’s practicality). In this way, I would say that the use of expert data should be ablated in the proposed method (so that we can evaluate how dependent the method is on this expert data). Another alternative is to also use expert data in the baseline methods (so that we can evaluate how much other methods also take advantage of this expert data).\n\n- The results from Figure 6 seem promising, but they are not analyzed/discussed in the paper. Indeed, the paper does not cite Figure 6 in the text in any circumstance.\n\n- There is no direct comparison between the proposed network architecture and that one from the prior method (Barreto et. al [1]). I understand that the prior method does not extend to the continuous case, but this ablation regarding the network in the proposed SAC framework will help understand how the proposed architecture improves transferability/generalization. \n\n- In section 3.2, the paper states that a central question for its scope is “Do SFs introduce implicit composability at the policy level?”, but I could not find any analysis/evaluation for this. The proposed experiments do not evaluate such property, and we could not conclude whether it emerges from the representation or not.\n\n- The paper lacks some details in the experimental setup (how to collect this expert data, number of random seeds, the dimensionality of  $\\psi$ for each task, number of network layers, how the goal is perturbed in the Reacher task, what is exactly the input for the task component in each scenario), making it difficult to reproduce the results. This information should be added to the paper (perhaps through the appendix). I also would like to invite the authors to release the source code for the ACSF implementation and for the experiments if this is possible (not required), which will address all the issues related to reproducibility.\n\n- Figure 7 shows that the proposed method performs on par with a goal-conditioned policy, as described in the text. However, it is not clear what is the intention behind this experiment: if they perform equivalently, what makes the use of successor features a good idea? It would be simpler to use the goal-conditioned baseline if there is no advantage.\n\nMinor Concerns:\n- The paper introduces some symbols in the text without any definition (for example, $\\phi$, $\\psi$, $\\boldsymbol{w}$). It makes the text harder to follow in the first read.\n- In section 3.2.2, “We forumlate” -> “We formulate”\n- In section 7, Fig 7c is cited twice, but the first time should be Fig 7b (door close task)\n- In Table 1, the number of layers seems very high (1024, 64, 1024). Perhaps it refers to the dimension of these layers?\n\nAdditional Suggestions:\nThere are additional suggestions. They are not necessary for acceptance, but in my perspective, they could improve the evaluation of the proposed method.\n\n- I recommend introducing other Representation Learning baselines as comparison methods. There are other works with different representation mechanisms [3,4,5] and such comparison will help validate the use of SFs for representation in the continuous multi-task setting.\n- Although is implicit that the proposed method improves the inference problem from the prior method [1], a proper evaluation (mathematical or experimental) would help understand it objectively  (i.e., “how much” it improves). The paper claims that this is the main contribution of the method, but this is not highlighted during the evaluation.\n- An interesting way to evaluate the final representation is to plot them in a lower-dimensional space. I would suggest plotting these representations (dynamics and task-specific) to evaluate interpretability and disentanglement visually.\n\nReferences:\n\n[1] André Barreto, Shaobo Hou, Diana Borsa, David Silver, Doina Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences, 2020.\n\n[2] Chen Ma, Dylan R. Ashley, Junfeng Wen, Yoshua Bengio. Universal Successor Features for Transfer Reinforcement Learning. CoRR, abs/2001.04025, 2020. URL http://arxiv.org/abs/2001.04025\n\n[3] Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, Martin Riedmiller. Learning an Embedding Space for Transferable Robot Skills. International Conference on Learning Representations, 2018.\n\n[4] Amy Zhang, Harsh Satija, Joelle Pineau. Decoupling Dynamics and Reward for Transfer Learning. CoRR, abs/1804.10689, 2018. URL http://arxiv.org/abs/1804.10689.\n\n[5] Dibya Ghosh, Abhishek Gupta, Sergey Levine. Learning Actionable Representations with Goal-Conditioned Policies. CoRR, abs/1811.07819, 2018. URL http://arxiv.org/abs/1811.07819.\n\n\n\n", "summary_of_the_paper": "The paper proposes a method to incorporate Successor Features (SFs)  in domains with continuous state and action spaces. It proposes an actor-critic architecture (a variation of the Soft Actor-Critic method) that learns disentangled representations for the environment dynamics and the tasks. The network architecture guarantees such disentanglement with two independent modules: one for the representation of the dynamics $\\phi$ (fed by the current state, action, and next state) and one for the task representation $\\boldsymbol{w}$ (fed by task-specific information, such as the goal). These modules are learned jointly in a single-stage training procedure, contrasting prior work [1]. The main contribution of this model is the enablement of the SFs for continuous domains without relying on the costly inference mechanism from the classic SFs framework implementation while enabling generalization among similar tasks.", "summary_of_the_review": "Although the paper is well motivated and presents interesting implementation details, the proposed method misses the discussion of concrete similar work. This related work makes the proposed method seem fairly incremental and only marginally novel. Furthermore, the evaluation setup presents some flaws that make it hard to validate the method and its limitations. Therefore, my perspective is that this paper does not meet the requirements for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an actor critic type of method consisting of two types of features -- dynamics and tasks, in the multi-task continuous control setting. While the topic of the research is interesting and relevant to ICLR, the reviewers have concerns with the novelty and technical significance of the work. Specifically, the proposed method is very similar to several other works leading to an incremental novelty. In addition, the method is evaluated only on simple environments. The concerns remain after the discussion period. 

In the next version of the manuscript, the authors are encouraged to pursue more difficult settings and modify the method to work on those problems. That would make the paper stronger, and lead to a more novel method evaluated on harder problems.
Decision: Reject