Composing Task Knowledge With Modular Successor Feature Approximators
Authors: Wilka Torrico Carvalho
Angelos Filos
Richard Lewis
Honglak Lee
Satinder Singh
Venue: ICLR 2023 poster (2023)
Keywords: deep reinforcement learning
successor features
generalization
compositional generalization

Abstract: Recently, the Successor Features and Generalized Policy Improvement (SF&GPI) framework has been proposed as a method for learning, composing and transferring predictive knowledge and behavior. SF&GPI works by having an agent learn predictive representations (SFs) that can be combined for transfer to new tasks with GPI. However, to be effective this approach requires state features that are useful to predict, and these state-features are typically hand-designed. In this work, we present a novel neural network architecture, “Modular Successor Feature Approximators” (MSFA), where modules both discover what is useful to predict, and learn their own predictive representations. We show that MSFA is able to better generalize compared to baseline architectures for learning SFs and a modular network that discovers factored state representations.



Published as a conference paper at ICLR 2023
COMPOSING TASK KNOWLEDGE WITH MODULAR SUCCESSOR FEATURE APPROXIMATORS
Wilka Carvalho*:
Angelos Filos?
Richard L. Lewis'
Honglak 1.3
Satinder Singh'
University of Michigan
University of Oxford
LL AI Research
ABSTRACT
Recently, the Successor Features and Generalized Policy Improvement (SF&GPI) framework has been proposed as method for learning, compos- ing, and transferring predictive knowledge and behavior. SF&GPI works by having an agent learn predictive representations (SFs) that can be combined for transfer new tasks with GPI. However, be effective this approach requires state features that are useful to predict, and these state-features are typically hesd-designed. In this work, we present a novel ncural net- work architecture, "Modular Successor Feature Approximators" (MSFA), where modules both discover what is useful predict, and learn their own predictive representations. We show that MSFA is able to better gen- eralize compared to baseline architectures for learning SFs and modular architectures for learning state representations.
1 INTRODUCTION
Consider a household robot that needs learn tasks including picking up dirty dishes and cleaning up spills. Now consider that the robot is deployed and encounters table with both a spill and a set of dirty dishes. Tdeally this robot can combine its training behaviors to both clean up the spill and pickup the dirty dishes. We study this aspect of generalization: combining knowledge from multiple tasks.
Combining knowledge from multiple tasks is challenging because it is not clear how to synthesize hit the behavioral policies or the value functions learned during training. This challenge is exacerbated when agent also needs to generalize novel appearances and environment configurations. Returning to our example, our robot might need to additionally generalize to both novel dirty dishes and to novel arrangements of chairs.
Successor features (SFs) and Generalized Policy Improvement (GPI) provide a mechanism to combine knowledge from multiple training tasks (Barreto et al. 2017 2020 SFs are predictive representations that estimate how much state-leatures (known as "cumulants") will be experienced given behavior. By assuming that reward has linear relationship between cumulants and a task vector, an agent can efficiently compute how much reward it can expect to obtain from given behavior. If the agent knows multiple behaviors, it can leverage GPI to compute which behavior would provide the most reward (see Figure 2 for an example). However, SF&GPI commonly assume hand-designed cumulants and don't have mechanism for generalizing to novel environment configurations.
Modular architectures are promising method for generalizing to distributions outside of the training distribution (Goyal et al. 2019 Madan el. al. 2021). Recently, Carvalho el al. (2021a) presented "FARM" and showed that learning multiple state modules enabled generalization to environments with unseen environment parameters (e.g. to larger maps with more objects). In this work, we hypothesize that modules can further be leveraged to discover stato-features that useful to predict.
"Contact author: wearhallh@uiich.eh

---

Published as a conference paper at ICLR 2023
(1) FARM: Multiple State Modules
(2) USFA: Monolithic SF Architecture
Ours: Moduler Successor
Feature Approximators
Behavior
Behavfor
Behawlor
State
State P
Q-veluea
81
8
11
Discovers a useful feeturee predict
Figure 1: (1) FARM learns multiple state modules. This promotes generalization to novel environments. However, it has mechanism for combining task solutions. (2) USFA learns a single monolithie architecture for predicing SFs and can combine task solutions. However, it relies on hand-designed state features and has no mechanism for generalization environments. (3) We combine the benefits of both. We leverage modules for reward-driven discovery of state features that are useful predict. These form the basis of their own predictive representations (SFs) and enables combining task solutions in novel environments.
novel
We present "Modular Successor Feature Approximators" (MSFA), a novel neural network for discovering, composing, and transferring predictive knowledge and behavior via SF&GPI MSFA is composed of a set of modules, which cach learn their own state-fcatures and corresponding predictive representations (SFs). Our core contribution is showing that an inductive bias for modularity can enable reward-driven discovery of state-features that are useful for zero-shot transfer with SF&GPI. We exemplify this with a simple state- feature discovery method presented in Barreto et al. (2018) where the dot-prot not-product between seate-features and a task vector is regressed to environment reward. This method enabled transfer with SF&GPI in continual learning setting but had limited success in the zero- shot transfer settings we study. While there are other methods for state-feat feat ure discovery, they add training complexity with mutual information objectives (Hansen et al. 2019) or meta- meta-gradients (Veeriah et al. 2019]. With MSFA, by adding only an architectural bias for modularity, we discover state-features that (1) support zero-shot transfer competitive with hand-designed features, and (2) enable zero-shot transfer in visually diverse, procedurally generated environments. We are hopeful that our architectural bias can be leveraged with other discovery methods in future work.
RELATED WORK ON CENERALIZATION IN RL
Hierarchical RL (HRL) is one dominant approach for combining task knowledge. The sequentially combine policies in time by having a "meta-policy" that sequentially activates "low-levol" policics for protracted periods of time. By leveraging generalize to longer instructions (Oh el al. 2017 Corona et al. 20200) to new instruction orders (Brooks el al. 2021), and to novel subtask graphs (Sohn et al. 2020 2022). We differ in That locus combining policies concurrently in time as opposed to sequentially in time. To do so, we develop modular neural network for the SF&GPI framework.
basic idea is that
hand-designed pre-trained low-level policies,
SFs are predictive representations that represent the current state as summary of the successive features to follow (see $3 for formal definition). By combining them with Generalized Policy Improvement, rescarchers have shown that they can transfer behaviors across object navigation tasks (Borsa continuous control tasks (Hunt et al. 2019/, and within HRI Tramework (Barreto et al. 2019 However, these works tend to require hand-designed cumulants which are cunbersome to design for every new environment. In our work, we integrate SE's with Modular RL to facilitate reward-driven discovery of cumulants and improve successor feature learning.
2019 Zhang et al.) 2017 Zhu al. 2017). across
Modular RL (MRL) (Russell & Zimdars, 2003) is framework for generalization by combining value functions. Early work dates back to (Singh 1992). who had a mixture- of-experts system select between separately trained value functions. Since then, MRL has been applied to generalize across robotic morphologies (Huang et al. 2020), to novel task-robot. combinations (Devin et al. 2017 Haarnoja et. al. 2018 and to novel language
2

---

Published as a conference paper at ICLR 2023
instructions Logeswaran et al. 2021 MSFA, is the first to integrate MRL with SF&GPI. This integration enables combining task solutions in novel environment configurations.
Generalizing to novel environment configurations with modules. Goyal et al. (2019) showed that leveraging modules to learn a state function improved out-of-distriution generalization. CCrvalhoo al. (2021a) showed that modified attention mechanism led to strong generalization improvements with RL. MSFA differs from both in that it employ modules for learning value functions in the form of SFs. This enables principled way to compose task knowledge while additionally generalizing to novel environment configurations.
3 PROBLEM SETTING AND BACKGROUND
We study a reinforcement learning agent's ability to transfer knowledge between tasks in an environment. During training, the experiences Merain tasks Mirsia = {Mr sampled from training distribution Persin(M). During testing, the agent is evaluated on Nestt tasks, {M&beer Puet sampled from a testing distribution Prest (MM. Fach task M, is specified as Partially Observable Markov Decision Process (POMDP). M, = (S", X, R. p. Pz). Here, S", A and X' are the environment state, action, and observation spaces. p(-1sf, ae) specifies the next-state distribution based on taking action ag in state sf, and fa (65) maps the underlying environment state to observation 21. We focus on tasks where rewards paramcterized by task vector w, ie. ri" = R(s dz. $441, w) is the reward obtained for transition (sf, ag, 8(41) given task vector Since this POMDP, state function that maps histories to agent state representations We do so with a recurrent function: 8t so (2 81-1, Given this learned state, want to obtaín behavioral policy (s;)) that best maximises the expected reward it will obtain when taking an action a a state 9y Q7 = QT."" (5,, az) ) E, I2 7fr["].
need to learn
Computing Successor Feature
Transfer via Generalized Policy Improvement
E, yd2 dddss ...
Legend
=s*) max m Trsts Tracs
Task: Open Fridge
Features
V Tiriige
Apple
3 Eiriage
Tzriage Waeta
Milk
Module
Mai1k
paramotors
Task: Open Drawer
Tailk
Rg Vdrazer
Task
Marass
Task Policies 77 fridde Wrraver
Wailk
chr
Cumulants agent experiences over time
Figure 2: High-level diagram of how MSFA be leveraged for transfer with SF&GPI. During training, we can have the agent learn policies for tasks e.g. "open drawer" and "open fridge". Each task leads the agent to experience different aspects of the anvironment-c.pp e.g. a "fork" during "open drawer" or "apple" during "open fridge". We can leverage MSFA to have different modules learn different "cumulants", Φ, and SFs, th. For example, module 1 (01) form the SF for policy. When the agent wants lo transfer its knowledge to test task e.g., "get milk" it can compute Q-values for that task as dot-product with the SFs of each training task. The highest Q-value then used to select actions.
estimate SFs for apple cunulants. Module SI's are combined to
Transfer with SF&GPI In order to leverage SFs Barretoo et al. 2017). one assumes agent has access to state features known "cumulants" Φt 018, Ct, 8 Given
3

---

Published as a conference paper at ICLR 2023
behavioral policy r(als), SFss are type of value function that use de pseudo-rewards
of =5 (5 oz) 1BTT
Σ
r pooti
(1)
If reward is (approxinately) rh" o then action-values can be decomposed Q vi w. This is interesting bocause it provides easy way to reuse task-agnostic features 4/ for new tasks.
We can re-use the SFs we've learned from training tasks Mirrei for transfer with GPL. Assume we have learned (potentially optimal) policies ) and their corresponding SFs (s) a) Mizate Given test task Wtest; we obtain new policy with GPT in two steps: (1) compute Q-values using the training task SFs (2) select actions using the highest Q- Q-value. This operation is summarized as follows:
T (8;i Wrest) E arg max max aeAA
(QTh. arg max aEA
max Refas...,eesss
Wrest)
(2)
:E(1I...,Pareii)
This useful because the GPI theorem states that will perform well all of the training policies, i.c. that @F (s,a) a) 2 max; Q (s, a)V(s, a) E (S x A) (Barreto et al. 2017).
SF&GPI enable transfer by exploiting structure in the RL problem: policy that maximizes a value function is guaranteed to perform at least as well as the policy that defined that value function. However, SF&GPI relies on combining fixed set of SFs. Another form of transfer comes from "Universal Value Function Approximators" (UVFAs) (Schaul el. al. 2015), which add the task-vector w as parameter to Q-approzimator parameterized by 0, Co(s, a, w). If Qo is smooth with respect to w, then Qo should generalize to test tasks nearby to train tasks in task space. Borsa al. (2019) showed that one could combine the benefits of both with "Universal Successor Feature Approximators", Since rewards r"; and therefore task vectors w, reference deterministic task policies Tw, one can parameterize successor feature approximators with task-vectors viTh = v/"" as Vo0(s, a, w). However, USFA assumed hand-designed cumulants. We introduce an architecture for reward-driven discovery of cumulants and improved function approximation of universal successor features.
4 MODULAR SUCCESSOR FEATURE APPROXIMATORS
We propose a new architecture Modular Successor Feature Approrimators (MSFA) for approximating SIs, shown in Figure 3 Our hypothesis is that learning cumulants and SI's with modules improves zero-shot composition of task knowledge with SF&GPI. MSFA accomplishes this by learning state modules {sa, 1k=1 that evolve with independent parameters 0g and have sparse inter- module information flow. MSFA then produces modular cumulants (0(4))p J==1 and SFs ( === by having their computations depend only individual modulc-states. For example, a cumulant may correspond to information about apples, and would be function only of the module representing state information related to apples. This is in contrast to prior work, which learns a single monolithic prediction module for computing cumulants and SF's (see Figure 3).
The rest of section is structured as follows. In section $4.1 Feature Learning within the Modular RL framework. We then describe our architecture, MSFA, for learning modular successor features in $4.2 In 44.3. we describe how to generate bchavior with MSFA. Finally, we describe the learning algorithm for MSFA in section 4.4
we derive Modular Successor
4.1 MODULAR SUCCESSOR FEATURE LEARNING
Following the Modular RL framework (Russell Zimdars 2003 we assume that reward
has an additive structure R(s;, az St+1) Σn R (st, at S(41) Σ, (* where Ra is the reward of the k-th module. We enforce that every module decomposes reward into an inner-product between its own task-description w(()) and task-agnostic cumulants ((k) E R:
R2(() (((h) (k) Here, simply break up the task vector into n pieces individual
4

---

Published as a conference paper at ICLR 2023
Universal Successor Feature Approximator (USFA)
Modular Successor Feature Approximator (MSFA)
Q. 1 Weant
4/
F1
MLP
MLP
MLP
MLP
MLP
Werain
14 Crain
Attention
Z
Legend Shared
Observation Encoder
Learning Cumulants 10z
Ooservation Encoder
Parameters
d
oz
(n)
Soparate
Parameters
[w
(n))
Recurrence
3t
min
||r""
di
wil
Lt
Figure 3: Left: Universal Successor Feature Approximator (USFA) learns single, monolithic successor feature estimator that uses hand-designed cumulants. Right: Modular Successor Feature Approximator (MSFA) learns set of successor feature modules, each with their own functions for (a) updating module-state, (b) computing cumulants, and (c) estimating successor features. Modules then share information with an attention mechanism. We hypothesize that isolated module computations facilitate learning cumulants that suppot generalization with GPI.
modules are responsible for subsets of the task vector. This allows us to decompose the action-value function
ai, w) Σ (st ae, wo (x))
Σ w (s1, w())
(3)
where we now have modular SFs ( (s, ) (see the Appendix for a derivation). Rather than band-designing modules or cumulants, aim to discover them from the environment reward signal.
4.2 ARCHITECTURE
We learn set of modules with states S, {s; (k)
1=11 They update at each time-step with and information from other modules
the observation It, the previous modulc-state
1 S,-)). Following prior work (Santoro et al. 2018 Goyal et al. 2019 Carvalho et al. 2017 with gating
2021ah, we have As combine transformer-style attention Vaswani mochanism (Parisotto et al.) 2020) to enforce that intor-modulo interactions are sparse. Since Ag is not the main contribution of this paper, describe these computations in more detail with our notation the Appendix. We summarize the high-level update below.
8 (k) se, (It, s (m
(k)
(4)
We learn modular cumulants and SFs by having sets of cumulants and SF's depend on individual modulc-states. Module cumulants depend on the modulc-state from the current and next time-step. Module SFs depend on the current module-state and on their subset of the task description. We summarize this below:
x(k) = dn (s) at 8441)
J = -/o 0t, (k))
(5)
We highlight that cumulants share parameters but differ their input. This suggests that the key is not having cumulants and SFs with separate parameters but that they are functions
5

---

Published as a conference paper at ICLR 2023
of sparse subsets of state (rather than all state information). We show evidence for this hypothesis Figure 6
We concatenate module-specifie cumulants and SF' to form the final outputs: o
Ju.n Note that cumulants, are only ( ................................ ot 7(m) and v/t"" (8t, dz, w) -w.1 ......
used during learning, update with the module-state from the next time-slep.
4.3 BELAAIII
During training, actions are selected in proportion to Q-vaues computed using task SFs as (5t, w) x Q(s,, a, w) = ve( a, w) w. In practice we use an apsilon-greedh policy, other choices. During testing, compute policies with GPI as (8t, Wreat) arg max, maa (80, Wteat), where M are train lask vectors.
though
4.4 LEARNING ALGORITIM
MSFA relies on three losses. The first loss, Lq, is a standard Q-lcarning loss, which MSFA uses learn optimal policies for the training tasks. The main difference here that MSFA uses particular parameterization of the Q-function QTx:"" a) = " a) T w. The second loss, Lw; is an SF learning loss, which we use as regularizer to enforce that the Q-values follow the structure in the reward function rt = of w. For this, we again apply standard Q- Q-learning but using SF's as value functions and cumulants pseudo-rewards. The final loss, Lo is a loss for learning cumulants that grounds them in the environment reward signal. The losses are summarised follows
L = ||+e + 7 a', w) w coo(se, ae, w) wwl
(6)
C. 11 + ( w) (or a,, w)l
(7)
C. = ||r2" oe w|
(8)
where a' arg max, (0( a, w) Tww w. Selecting the next action via the combination of all modules ensures they individually convergence to optimal values (Russell & Zimdars 2003)
The final loss is f == aqla af + asLas
5 EXPERIMENTS
(a) BabyAl
(b) Procgen
(c) Minihack
Figure 4: We study an agent's ability to combine task knowledge in three envi- ronments. (a) In AI,y agent learns to pick up object type at time during training. During testing, the agent must pickup combinations of object types while avoiding other object types. This the setting used by USFA which assumed hand-designed cumulants. (b) In Procgen, we study extending this form of generalization to visually diverse, procedurally generated environment. (c) In Minihack,
go beyond combining object navigation skills. Here, an agents needs to combine (1) avoiding teleportation traps, (2) avoiding monsters, and (3) partial visibility around the agent.
We study generalization when training behaviors must be combined concurrentyy in time in the presence of novel object appearances and layouts. The need to combine training behaviors

---

Published as a conference paper at ICLR 2023
tests how well MSFA can leverage SF&GPI. Generalization novel object appearances and layouts tests how well MSFA's modular construction supports generalization to novel environment configurations.
Baselines. (1) Universal Value Function Approximator (UVFA) (chaul et al. 2013
which takes the task imput: Qo w). This comparison shows shows the transfer benefits of SF&GPI. (2) UVFA with Foature-Attending Recurrent Modules
( w). Each state-factor g( is the outpul of FARM module. (3) Modular Value Function Approximator (MVFA) is
instead takes Maate-fcttors as input
adaptation of jaarroja et al. 2018) where modules learn individual Q-values 2 (s(0, w(l)). Comparing to modules for learning value functions in the form of SFs. (4) Universal Successor Function Approximator (USFA) (Borsa al. 2019) leverages a single monolithic function for successor features with hand-designed cumulants. USFA is an upper-bound baseline that allows us to test the quality cumulants and successor features that MSFA learns. We also test variant of USFA with learned cumulants, USFA-Learned-g) which shows how the architecture degrades without oracle cumulants.
and MVFA enables to study the benefits of leveraging
We implement the state modules of MSFA with FARM modules (Cavaahho et al. 2021b)- For UVFA and USFA, we learn stale representation with an LSTM (Hochre iter & Schmidhuber 1997p. We implement all o, b, and functions with Mutli-layer Perceptrons. We train UVFA and UVEA+FARM with n-step Q-learning (Watkins & Dayan 1992). When learning cumulants, USFA and MSFA have the exact same losses and learning alogirthm. They both learn Q-values and SFs with n-step Q-lcarning. We use n = 5. When not learning cumulants, following (Borsa et al.) 2019)) USFA only learns SFs with n-step Q-learning. All agents are built with JAX (Bradbury et al. 2018) using the open-source ACME codebase (Hoffman al. 2020 for reinforcement learning.
Implementation.
5.1
COMBINING OBJECT NAVIGATION TASK KNOWLEDGE
1,1,0,0 (1.00)
1,1.5,.5 (1.22)
1,1,1,1 (1.73)
5,,1,-55-55 (0.87)
45
Episode Return
30
45
30
30
15
15
01
-10
-1,1,0,1 (1.41)
-1,1,-1,1 (1.73)
-1,1, (1.73)
45
Episade Retur
30
BFS State Orade
UVFA
30
UVFA FARM
MVTA
25-
10
USFA (Leerned d)
(Learned
USFA toracle #1
0M
2M
4M
0M
2M
4M
0M
2M
4M
Frames
Frames
Frames
Figure 5: MSFA matches USFA, which has hand-designed cumulants. We show mean and standard error generalization episode return across 10 runs. We put a task's 's L2 distance to the closest train task in parenthesis. USFA best generalizes to novel combinations of picking up and avoiding objects. Once USFA learns cumulants, its performance degrades significantly. UVFA-based methods struggle as more objects should be avoided or tasks are further in distance to train lasks.
MSFA learns modular functions for computing & and estimating th. We hypothesize that this facilitates learning cumulants that respond to different aspects of the environment (e.g. to different object categories). This leads to the following research questions. R1: Can we recover prior generalization results that relied object categories? R2: How important is it to learn modular functions for & and ø? R3: Without GPI, do learning modular functions for and s) still aid in generalization?
hand-designed cumulants for different

---

Published as a conference paper at ICLR 2023
Setup. We implement simplified version of the object navigation task of (Borsa et al. 2019; in the Shevalier-Boisvee et al. 2019) BabyAl environment. The environment consists of 3 instances of 4 object categories. Observations the agent can rotate left. or right, move forward, or pickup an object. When it picks up object, following (Borsa et al., 2019), the object is respawned somewhere the grid. Task vectors lie in w E IR with training tasks being the standard unit vectors, For example, [0, 1, O, 0] specifies the agent must obtain objects of category 2. Generalization tasks are linear combinations of training tasks. For example, [-1, 1. 1.1..11. 1] specifies the agent must obtain categorics and while avoiding categorics and [orsa al. (2019) showed that USFA could generalize with hand-designed cunulants that described whether an object was picked up. We describe challenges for this lask in detail the Appendix.
partial and egocentric. Actions:
RI: MSFA is competitive with USFA, which uses oracle Φb Figure shows USFA with a similar generalization trend to (Borsa et al. 2019.. Tasks get more challenging as they are further from train tasks or involve avoiding more objects. For simply going to combinations of objects, USFA-Learued-> does slightly worse than MSFA. However, with more objects to avoid, all methods except MSFA (including USFA-Learned-o) degrades significantly. For comparison, we show performance by an oracle bread-first-seareh policy with access to ground-truth state (BFS State Oracle). All methods have room for improvement when objects must. be avoided. In the appendix, we present heat-maps for how often object categories were picked up during different tasks. We find that MSFA most matches USFA, while SSSFA-Learned-g commonly picks up all objects regardless of task.
(8) Ablation of Modularity
(b) Ablation Generalized Policy Improvement
-1,1,0.1
-1,1,111
1111111
111.111
Epicil Retur
30
12
20
30
10
014
2H
4M
071
2H
611
DN
214
6M
09
24
4N
015
24
4N
24 hames
4M
Rames
model 0 mod--p
mos-- + Racc
1at-0 mcde
flc-0
Teb
1256 mLC OF
US94 Learies W0 OM
(594 Koock WO
Figure 6: Learning modular and s is key to generalization and improves generalization even without GPI. We show mean and standard error generalization episode return across 10 runs. (a) We ablate having modular functions for de and tho- Generalization results degrade significantly. (b) We ablate leveraging GPT for generalization from all SF-based methods. MSFA without GPT outperform both USFA-Learned-o with GPI and USFA without GPI. This shows the utility of modularity for generalization.
R2: Learning modular & and ve functions critical for generalization. Learning an entangled function corresponds to learning monolithic function for v & where
concatenate module-states, e.g. Teth ( ..... s(x), a, w,w.). Modular functions correspond to equation 5] Figure 0 (a) shows that without modular functions for é and v! performance
severely degrades. This also highlights that a naive combination of USFA SIPA-FAR FFARM-with entangled functions for é and to does not recover our generalization performance.
R3: Modularity alone improves generalization. For all SF-based methods, we remove GPI and select actions with a greedy policy: T(s) = arg max, vo((t, a, w) w. Figure (b) shows that GPI is critical for generalization with USFA as expected. USFA-Learned-a benefits less from GPI (presumably bocause of challenges in learning Φ). Interestingly, MSFA can generalize relatively well without GPI, sometimes doing better than USFA without GPI.
5.2 COMBINING OBJECT NAVIGATION TASK KNOWLEDGE WITH NOVEL APPEARANCES AND ENVIRONMENT CONFICURATIONS
Beyond generalizing to combinations of tasks, agents will need to generalize to different layouts and appearances of objects. R4: Can MSFA enable combining task knowledge in a visually diverse, procedurally generated environment?
Setup. We leverage the "Fruitbot" environment within ProcGen (Cobbe et al. 2020 Here, agent controls paddle that tries to obtain certain categories of objects while avoiding
8

---

Published as a conference paper at ICLR 2023
others. When the agent hits wall fence, it dies and the episode terminates. If the agent collects a non-task object, nothing happens. Observations are partial. Actions: At each time-step the agent moves step forward and can move left right shoot pellets to open fences. Training and generalization tasks follow the same setup as 5.1
1,0,0,-1
1,0,0,1
1,0,1,1
1,1,1,1
80
Episode Return
1.5
0.0
45-
60
30
-1.5
30
-3.0
15
20
-4.5 0M
0
0
SM Frames
10M
0M
SM Frames
10M
0M
SM Frames
10M
0M
5M
10M
Frames
USFA &
MSFA
Figure 7: MSFA is able to combine task knowledge in a visually diverse, procedu- rally generated ProcGen environment. We find that no method is able to do well when there are objects to avoid (w = [1,0,0, -1]) in this setting (see text for more). However, as more objects need be collected MSFA best generalizes (10 runs).
R4: MSFA enables combining task knowledge of object navigation tasks in visually diverse, procedurally generated environment. Figure ( shows that when agent has to generalize to collecting more objects, modular architectures generalize best, with MSFA doing best. When objects have to be avoided, we see that no architecture does well, though MSFA tends to do better. We observe that avoiding objects leads agents to hit walls. Since the agent always moves forward at each time-step in Fruibot, this makes avoiding objects a particularly challenging type of generalization.
DISCUSSION AND CONCLUSION
We have presented "Modular Successor Feature Approximators", modular neural network for learning cumulants and SF's produced by their own modules. We first showed that MSFA competitive with prior object navigation generalization results that relied on hand-designed cumulants (5..1) Afterwards,
showed that MSFA improves agent's ability to combine task knowledge in a visually diverse, procedurally generated environment ($5.2). We also show that MSFA combine solutions of hetcrogencous tasks (S Our ablations show that learning modular cumulants and SFs is critical for generalization with GPI.
We compared MSFA to (1) USFA, monolithic architecture for learning cumulants and STs; (2) FARM, an archilecture which learns multiple stale-modules but combines them with monolithic Q-value function. Our results show that when learning cumulants, MSFA improves generalization with SF&GPI compared to USFA. Additionally, without GPI, MSFA as an architecture improves generalization as compared to both FARM and USFA.
Limitations. While we demonstrated reward-driven discovery of cumulants for transfer with SF&GPI, we focused on relatively simple task encodings. Future work can extend this more expressive encodings such as language embeddings. Another limitation is that we did not explore more sophisticated state-feature discovery methods such meta- geta-gradients (Veeriah) el al. 2019). Nonetheless, we think that. MSFA provides an import ant. insight. for future work: that modularity is a simple but powerful inductive bias for discovering state-fees feattres that enable zero-shot transfer with SF&GPI
Future directions. SI's are useful for exploration (Janz et al. 2019 Machado et al. 2020); for discovering and combining options (Barreto et al. 2019 Hansen et al. 2019] for transferring policies across environments (Zhang et al. 2017): for improving importance sampling (Fujimoto et al. 2021); and for learning polictes from other agents (Filos et al. 2021). We hope that future work can leverage MSEA for improved state-feature discovery and Se-leernnig in all of these settings.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported in part by a University of Michigan Rackham Merit Fellowship and grant from LG AI Rescarch. The authors would also like to thank Andrew Lampinen, Cameron Allen, and the anonymous reviewers for their helpful comments.
REFERENCES
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. NIPS, 30, 2017.
André Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In 1CML, pp. 501-510. PMLR, 2018.
André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Shibl Mourad, David Silver, Doina Precup, et al. The option keyboard: Combining skills in reinforcement lcarning. Advances in Neural Information Processing Systems, 32, 2019.
André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup. Fast reinforcement learning with generalized policy updates. PNAS, 117(48):30079 30087, 2020.
Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado Van Hasselt, David Silver, and Tom Schaul. Universal successor features approximators. ICI.R, 2019.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wandernan-Milng and programs, 2018. URL
Qiao Zhang. JAX: composable transformations of
bottp///githnn com.bbbb.coo/ggoge///aa
Ethan Brooks, Janarthanan Rajendran, Richard L Lewis, and Satinder Singh. Reinforcement learning of implicit and explicit control flow instructions. In International Conference on Machine Learning, pp. 1082 1091. PMLR, 2021.
Wilka Carvalho, Andrew Lampinen, Kyriacos Nikiforou, Felix Hill, and Murray Shanahan. Feature-attending recurrent modules for generalization in reinforcement learning. arXiv, 2021a.
Wilka Carvalho, Anthony Liang, Kimin Lee, Sungryull Sohn, Honglak Lee, Richard L Lewis, and Satinder Singh. Reinforcement learning for sparse-reward injecttinteraction tasks in first-person simulated 3d environment. IJCAI, 2021b.
Maxime Chevalier-Boisvert,, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform study the sample efficiency of grounded language learning. ar preprint arXiv: 082772, 2018
Maxime Chevalier-
Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Baby First steps towards grounded language learning with a human in the loop. In International Conference on Learning Representations, 2019. URL https://openreviiss wwt.:e/oooommmtddd.ee.ooooo
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp. 2048 2056. PMLR, 2020.
Rodolfo Corona, Daniel Fried, Coline Devin, Dan Klein, and Trevor Darrell. Modular networks for compositional instruction following. arXio preprind 2rx 021674, 2020.
10

---

Published as a conference paper at ICLR 2023
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 2169-2176. IEEE, 2017.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yolam Doron, Vlad Firoiu, Tim Harley, lain Dumning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416. PMLR, 2018.
Angelos Filos, Clare Lyle, Yarin Gal, Sergey Levine, Natasha Jaques, and Gregory Farquhar. Psiphi-learning: Reinforcement learning with demonstrations using successor features and inverse temporal difference learning. In International Conference Machine Learning, pp. 3305-3317. PMLR, 2021.
Scott Fujimoto, David Meger, and Doina Precup. A deep reinforcement learning approach to marginalized importance sampling with the successor representation. In International Conference on Machine Learning, pp. 3518 3518-3529. PMLR, 2021.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Beruhard Schölkopf. Recurrent independent mechanisms.. arXiv preprint arXiv:1909 10893, 2019.
Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Composable deep reinforcement learning for robotic manipulation. In 2018 IEFE international conference on robotics and automation (ICRA), pp. 6244 6251. IEEE, 2018.
Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde Farde- and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint 1906 05030, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, (8):1735-1780, 1997.
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, et al. Acme: A rescarch framework for distributed reinforcement learning. arXiv preprint or Xrx::20066.00 00979, 2020.
Wenlong Huang, Igor Mordatch, and Deepak Pathak. modular policies for agent-agnostic control. In International Conference on Machine Learning, pp. 1455-4464 PMLR, 2020.
One policy to control them all: Shared
Jonathan Hunt, Andre Barreto, Timothy Lillicrap, and Nicolas Heess. Composing entropic policies using divergence correction. In International Conference on Machine Learning, pp. 2911 2920. PMLR, 2019.
David Janz, Jiri Hron, Przemyslaw Mazur, Katja Hofmann, José Miguel Hernández-Lobato, and Sebastian Tschiatschek. Successor uncertainties: exploration and uncertainty in temporal difference learning. Advances in Neural Information Processing Systems, 32, 2019.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
Lajanugen Logeswaran, Wilka Torrico Carvalho, and Honglak Lee. Learning compositional lasks from language instructions. In Deep RL Workshop NeurIPS 2021, 2021.
Marlos C Machado, Marc G Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, number (4, pp. 5125-5133 2020.
11

---

Published as a conference paper at ICLR 2023
Kanika Madan, Nan Rosemary Ke, Anirudh Goyal, Bernhard Schölkopf, and Yoshua Bengio. Fast and slow learning of recurrent independent mechanisms. arXiv preprint arXiv:2105.088 08710, 2021.
Volodymyr Mnih, Koray Kavukeuoglı, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529--33 533, 2015.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In International Conference on Machine Learning, pp. 2661- 2670. PMLR, 2017.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascamu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International conference on machine learning, pp. 7487-7498. PMLR, 2020.
Stuart Russell and Andrew Zimdars. Q-decomposition for reinforcement learning agents. In Proceedings of the 20th International Conference on Machine Learning (ICMI-033, pp. 656-663, 2003.
Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Küttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning research. ar preprint 0rXiv:2109.11220 2021.
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. Advances in neural information processing systems, 31, 2018.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference machine learning, pp. 1312-1320. PMLR, 2015.
Satinder Pal Singh. Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning, 8(3):333-333 1-23-339,111 1992.
Sungryull Sohn, Hyunjae Woo, Jongwook Choi, and Honglak Lee. Meta reinforcement learning with autonomous inference of subtask dependencies. ar preprint ar xrXi::2001.002, 2020.
Sungryull Sohn, Hyunjae Woo, Jongwook Choi, Lyubing Qiang, Izzeddin Gur, Aleksandra Faust, and Honglak Lee. Fast inference and transfer of compositional task structures for few-shot task generalization. In Uncertainty in Artificial Intelligence, pp. PMLR, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. orx, 2017.
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L Lewis, Junhyuk Oh, Hado P van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. Advances in Neural Information Processing Systems, 32, 2019.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279 292, 1992.
Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep reinforcement learning with successor features for navigation across similar environments. In 2017 IEEE/ RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2371 2378. IEEE, 2017.
Yuke Zhu, Daniel Gordon, Eric Kolve, Dicter Fox, Li Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, and Ali Farhadi. Visual semantic planning using deep successor representations. In Proceedings of the IEEE international conference on computer vision, pp. 483-492,
2017.
12


Reviews:
[{"review_id": "pD0U-qVzmWB", "reviewer": "Reviewer_GLtt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of task generalization through the use of the successor features framework. It proposes using a modular architecture with the universal successor features method (USFA; introduced in prior work). The modular architecture enforces disentanglement by conditioning each module's predictions on only a subset of the other modules. The experiments show that such a modular architecture is indeed useful in improved task generalization when combined with a USFA algorithm variant which learns the cumulants.", "summary_of_the_review": "I really like how succinctly the paper conveys the main message (precisely Figure 5 and Figure 6 are enough to convince the efficacy of the proposed modular architecture). The paper is written well (barring a few confusions I have below around some of the definitions) and is simple to understand for the most part. Overall, I think it makes a useful contribution, with well-constructed experiments and clear writing. Hence, I vote for acceptance. \n\nIn Figure 3, $z_t$ is the output of the encoder, which, going by the text, should be denoted by $s_t$?\nWhy is $x_t$ provided as input in Eq 4? Shouldn’t $s_t$ already encode that information?\nA_theta is defined as a function of $s^k_t$ and $S_{t-1}$, but wouldn’t $S_{t-1}$ already include $s^k_t$, given how it’s defined in the first line of Section 4.2?", "strength_and_weaknesses": "Strengths\n\n- Well motivated problem\n- Simple proposed method\n- Succinct results\n\nWeaknesses\n\n- Parts of writing can be improved\n- Experiments are performed on a limited number of (simple/toy-like) domains ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eVGSdUwU4A", "reviewer": "Reviewer_kJTC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new architecture that is useful for generalization to new tasks, that may be previous combination of prior tasks. It adapts from existing literature on successor features and GPI, and shows that a modified architecture that predicts useful representations while also learning representations can be significant in terms of generalization. This work is an adaptation of recent line of works on modular architectures to achieve OOD generalization. \n\nOverall, there seems to broad claims that the proposed architecture learns compositional aspects of the tasks that helps with generalization to out of distribution tasks. \n", "summary_of_the_review": "I would argue for a rejection of this work mainly because of the broad claims in terms of \"disentanglement\" and \"compositionality\" which are not well justified from the results. The paper proposes a simple archiectural change, variations of which has been proposed in the past well. As such, the proposed work is not fully novel, yet there are broad claims which makes the paper look really fancy!", "strength_and_weaknesses": "\n\t- The main idea of this paper is to propose a new architecture that can achieve modularization, leading to better generalization. The paper achieves this by learning useful representations from such a modular architecture, in experiments with procedurally generated environments. \n\t- The paper considers a train test generalization setting where the agent is trained on a seqeuence of tasks to then test for generalization to new tasks. \n\t- The key idea is to use attention mechanism in the resulting architecture that can capture different aspects of the task. Given observations, and the encoded reprsentation Z, I want to understand more why these separate modules are actually useful? \n\t- I understand the intuition behind the work, but how these modules are actually capturing what the paper claims to achieve is not clear? What are the different components of Z_t that each of the module is trying to capture? Why would simply using an attention module be useful here? I am not sure how the intuitions are justified from this architecture?\n\t- The paper claims to achieve disentangled cumulants? How do you justify that? Is there a simple example demonstrating this?\n\t- Experimental results compare to existing baselines such as UVFA. However, there are other architectural modular networks such as RIMS (Goyal et al) that also proposed such architectures? Why are these baselines architectural changes not considered?\n\t- My biggest worry of this type of work is the claims the authors try to make, but the results are only in terms of returns or some existing evaluation metrics, which does not justify the claims? The paper provides lots of intuitions of why the architectural bias is important, in terms of disentangling cumulants in the context of successor features; where the architecture captures different aspects of the tasks that helps for generalization. However, empirical results do not fully support this, neither are there theoretical claims for it.\n\t- Why is the the term compositional exactly used here? How is this term justified? I think the paper makes a broad claim but fails to fully backup results for it\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bWI9VUhrhH", "reviewer": "Reviewer_xfgM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors tackled the problem of combining knowledge from multiple tasks. They introduced Modular Successor Feature Approximator (MSFA) as a new network architecture that both identifies what is important to predict and also learns the corresponding representation. From another lens MSFA removes the need to hand-design cumulants (state-dependent features that capture the return linearly) for the Universal Successor Feature Approximator work by Borsa et al. [ICLR-2019].\n", "summary_of_the_review": "Good submission with yellow flag on reproducibility.  ", "strength_and_weaknesses": "Strengths\n+ Paper is well-written with solid contribution in the space\n+ Generalization across tasks is a critical step towards practical reinforcement learning. Hence it is of great interest to the community\n+ Ablation studies are thorough showing the impact of modularity and generalization even in heterogeneous tasks\n\nWeaknesses\n- Reproducibility: code or algorithm were not discussed in the paper. While few parameters were covered, it is unlikely for someone to be able to reproduce the results.\n- The empirical results are done in toy domains. What are the main limits to apply MSFA to larger and more realistic tasks?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YncM76y6v7l", "reviewer": "Reviewer_H7je", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper authors propose a method to generalize agents better by combining knowledge from multiple tasks. While the agent should generalize on a single given task, it should be able to do so across multiple ones. The proposed method, MSFA, aims at solving this problem.", "summary_of_the_review": "See above.", "strength_and_weaknesses": "The idea behind MSFA which is to learn disentangled cumulants and successor features is very interesting and promising. Overall, the paper is theoretically and practically reasonable. One thing that is worth studying is how modules in the MSFA architecture are correlated with one another.\n\nIn the experiments section, it is not mentioned how many seeds were used to obtain the provided results. Clarification regarding that would be useful. On a similar note, according to results provided, 4 runs were used. In order to better validate the generalizability and robustness of the MSFA, running more tests are required.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster