Learning Successor Features the Simple Way
Authors: Raymond Chua
Arna Ghosh
Christos Kaplanis
Blake Aaron Richards
Doina Precup
Venue: NeurIPS 2024 poster (2024)
Keywords: deep reinforcement learning
representation learning
continual reinforcement learning
successor feature
successor representation
Primary area: reinforcement_learning
TL;DR: A simple approach for learning Successor Features from pixel-level observations for Continual Reinforcement Learning

Abstract: In Deep Reinforcement Learning (RL), it is a challenge to learn representations that do not exhibit catastrophic forgetting or interference in non-stationary environments. Successor Features (SFs) offer a potential solution to this challenge. However, canonical techniques for learning SFs from pixel-level observations often lead to representation collapse, wherein representations degenerate and fail to capture meaningful variations in the data. More recent methods for learning SFs can avoid representation collapse, but they often involve complex losses and multiple learning phases, reducing their efficiency. We introduce a novel, simple method for learning SFs directly from pixels. Our approach uses a combination of a Temporal-difference (TD) loss and a reward prediction loss, which together capture the basic mathematical definition of SFs.  We show that our approach matches or outperforms existing SF learning techniques in both 2D (Minigrid) and 3D (Miniworld) mazes, for both single and continual learning scenarios. As well, our technique is efficient, and can reach higher levels of performance in less time than other approaches. Our work provides a new, streamlined technique for learning SFs directly from pixel observations, with no pretraining required.

Learning Successor Features the Simple Way
Raymond Chua
Arna Ghosh
Christos Kaplanis
Blake A. Richards f
Doina Precup
Abstract
In Deep Reinforcement Learning (RL), it is a challenge to learn representations that not exhibit catastrophic forgetting or interference non-stationary environments. Successor Features (SFs) offer potential solution to this challenge. However, canonical techniques for learning SFs from pixel-level observations often lead to representation collapse, wherein representations degenerate and fail to capture meaningful variations the data. More recent methods for learning SFs can avoid representation collapse, but they often involve complex losses and multiple learning phases, reducing their efficiency. We introduce a novel, simple method for learning SFs directly from pixels. Our approach uses combination of Temporal- difference (TD) loss and a reward prediction loss, which together capture the basic mathematical definition of SFs. We show that our approach matches outperforms existing SF learning techniques in both 2D (Minigrid), 3D (Miniworld) mazes and Mujoco, for both single and continual learning scenarios. As well, our technique is efficient, and can reach higher levels of performance in less time than other approaches. Our work provides new, streamlined technique for learning SFs directly from pixel observations, with no pretraining required
Introduction
Deep reinforcement learning (RL) Sutton and Barto 2018 is important to modern artificial intelli- gence (AI), but standard approaches deep RL can struggle when deployed for continual learning Parisi et al. 2019 Hadsell et al. 2020 Khetarpal et al, 2022]. When either the reward function or the transition dynamics the environment changes, standard deep RL techniques, such as deep
O-learning, will either struggle to adapt to the changes or they will exhibit catastrophic forgetting
Kirkpatrick et al. 2017 Kaplanis et al. 2018). Given that the real-world often non-stationary,
better techniques for deep RL in continual learning are major goal in Al research Rusu et al. 2016
Rolnick al. 2019 Powers et al. 2022 Abbas et al. 2023 Anand and Precup 2024
One potential solution that researchers have explored is the use of Successor Features (SFs). Successor Features, the function approximation variant of Successor Representations (SRs) [Barreto al. 2017]. decompose the value function into separate reward function and transition dynamics representation Dayan 1993]. In doing so, they make it easier adapt changes in the environment, because the network can relearn either the reward function or the transition dynamics separately [Borsa et al.
2018
Barreto al. 2018 2020 Hansen al.) 2019 Lehnert and Littman 2019 Liu and Abbeel
2021
Furthermore, there are theoretical guarantees that SFs can improve generalization in multi-task
"Correspondence to: ray.r.chua@ @gmail.cmmmmommmmoo.o
1 Dept of Neurology & Neurosurgery, and Montreal Neurological Institute of McGill University.
1 "Co-senior Authorship. CIFAR Learning in Machines and Brains.
$ School of Computer Science, MeGill University & Mila
1 Google Deepmind
Cood: htt.t///se.ttttt.ooo//////oaaa/e
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Canonical-SF
Simple SF (ours)
Similarity learned representations Before After training
Episade Retums
1.0
(a) Center-Wall: Fully-Obs
(c) Cluster Quality Metrics: Silhouctte vs. DB
Cosine Similaity
Higher the beter
Divis-Rouldi Roul-i (D3B
Liveer the betar
Avg
0.5
0.8
8.6
0.8
Silboerte
0.4
0.4
02
$0.4
0.2
Steps
0.6
1.0
1e6
0.2
Canonical Simple
Cansiies
Sumple
0
SV
SF
Before
After
Before After
Figure 1: (a) Results from single task within 2D two-room environment, illustrating the suboptimal performance of the canonical Successor Features (SF) learning rule (Eq. 4 due to representation collapse. (b) In the canonical SF approach, the average cosine similarity between pairs of SFs converges towards a value of 1, demonstrating representation collapse occurs. (c) The canonical SF learning rule does not develop distinet clusters in its representations, as evidenced by lower silhouette scores and higher Davies-Bouldin scores, which again indicates representation collapse. A mathematical proof can be found in section 3.4
settings [Barreto al. 2017]. SFs are therefore promising candidate for deep RL in non-stationary settings.
However, learning SFs is non-trivial. The most straightforward solution, which is to use a temporal- difference (TD) error on subsequent observations [Barreto et al. 2018), can lead to representational collapse, where the artificial neural network maps all inputs to the same point in high-dimensional representation space. This phenomenon commonly observed in various deep learning pipelines that end up learning similar or identical latent representations for very different inputs [Aghajanyan et al. 2020]. In RL, representation collapse can lead to different states or state-action pairs being mapped to similar representations, leading to suboptimal policy decisions or inaccurate estimation of values.
To solve this problem, a variety of solutions have been proposed. One solution is to use an additional reconstruction loss (Kulkarni et al.) 2016 Zhang et al. 2017 Machado et al. 2020 to force the network to maintain information about the inputs in representations. Another solution is to use extensive pretraining coupled with additional loss terms to encourage high-entropy representations
Hansen et al. 2019 Liu and Abbeel 202T]. More recently, an alternative solution using loss terms to promote orthogonal representations has been put forward (Mahadevan and Maggioni 2007 Machado et al. 201 Finally, an unconventional approach integrates Q-learning and reward prediction losses with the SF-TD loss, enhancing the learning process by providing additional supervisory signals that improve the robustness and effectiveness of the successor features [Janz et al. 2019]. This method allows the network to simultancously learn the basis features, successor features, and task encoding vector, with the hope that the learned variables will satisfy their respective constraints.
Though these solutions prevent representational collapse, they can impair learning. introduce addi- tional training phases, or add expensive covariance calculations to the loss function [Touati et al. 2022). Ideally, there would be way to learn deep SF' directly during task engagement with a simple, easy to calculate loss function.
Here, we introduce a simple technique for learning SFs directly during task engagement. We designed a neural network architecture specifically achieve this training objective. Our approach leverages the mathematical definition of SFs and constructs loss function with two terms: one that Iearns the value function with a TD-error, and another that enforces representations that make the rewards linearly predictable. By mathematical definition, this loss is minimized when the system has learned set of SFs. We show that training with this loss during task engagement, facilitated by our neural network architecture, leads to the learning of deep SFs as well as, or better than, other approaches. It does so with no pretraining required and very minimal computational overhead. As well, we show that our technique improves continual reinforcement learning in dynamic environments, in both 2D grid worlds and 3D mazes. Altogether, our simple deep SF learning approach is an effective way to achieve the benefits of deep SFs without any of the drawbacks.
2 Related work
Our work builds on an extensive literature on decomposing the value function dating back to the 1990s DDaan 1993). More recent work on learning deep SFs falls broadly into three categories

---

of solutions. The first are solutions that use a reconstruction term in the loss function in order to avoid representational collapse (Kulkarni al. 2016 Zhang et al., 2017 Machado et al. 2020). This general approach is effective at avoiding collapse, but it can lead to impaired performance on the actual RI. task, as we show below. The next set of solutions rely on hand-crafted features (Lehnert
et al.
2017, Barreto et al. 20188, Borsa et al. 2018 Madarasz and Behrens 2019] Machado et al.
2021
Smukpere et al.) 2021| Nemecek and Parr 2021 Brantly et al. 202 McLeod et al. 2022
Alegre et al. 2022 Reinke and Alameda-Pineda 2021 or hand-crafted task knowledge [Hansen et al.
2019 Filos et al. 2021| Liu and Abbeel [22 I Carvalho et al. 2023a] In these cases, the networks
learn and generalize well, but hand-crafiee solutions cannot scale-up real-world applications. Another category of solutions uses pretraining of the features in the deep neural network before any
engagement with the actual RL task [Fujimoto et al. 2021 Abdolshah et al. 202T Touati et al. 2022
Carvalho et al. 2023b]. Such solutions are not as applicable for continual RL because they introduce the need engage in new pretraining when the environment changes, which assumes some form of oracle knowledge of the environment. Finally, there are solutions that rely on additional loss terms to encourage orthogonal representations, since SRs are built off of purely orthogonal tabular inputs (Touaii et al., 2022 Farebrother et al. [023]. These techniques can improve SF learning, but they require computationally expensive calculations of orthogonality in the basis features.
Among these prior approaches, the work most closely related to ours is the application of multiple losses jointly learn the SFs, task- task-encoding vector, and Q-values [Ma et al. 2020]. However, there are several key differences: (1) Our approach does not require the agent to be provided with a goal-it is learned through interaction with the environment; (2) We provide direct evidence that our method works with pixel inputs; (3) We demonstrate that our approach eliminates the need for an SF loss; and (4) By removing the SF loss, we reduce the number of hyperparameters required, thereby simplifying the model.
In our results below, we compare our method to these classes of solutions described, namely re- construction solutions [Machado et al. 2020), pretraining solutions [Liu and Abbeel 202T]. and orthogonality solutions Touati et al. 2022
3 Preliminaries
3.1 Reinforcement Learning
The RL setting is formalized as Markov Decision Process defined by a tuple (S, p, r, 7), where S is the set of states, is the of actions, r : E is the reward function, p : S x A [0, 1] is the transition probability function and y E 10,1) 1) is the discount factor which is being to used to balance the importance of immediate and future rewards (Suuton and Barto 2018).
At each time step I, the agent observes state S, E S and takes an action A, E A sampled from a policy S x A [0, 1). resulting in to transition of next state S,+1 with probability p(Se-1 St A.) and the reward Re+1-
3.2 Successor Features
SFs are defined via a decomposition of the state-action value function (i.e. the expected return), Q, into the reward function and a representation of expected features occupancy for each state S, and action A, of time step l:
As, = A,, w) w
where v E Rn are the SFs that capture expected feature occupancy and w E R" is a vector of the task encoding, which can be considered a representation of the reward function [Borsa al. 2018
Canonically, the SFs for a state-action pair (s, a) under a policy π are defined as:
(s, ④) = E
1 2141 S: s, A,
where Φ E R is set of basis features, and is the policy Barreto et al. 2017
3

---

However, shown by Borsa et al. 2018 we can treat the task encoding vector w as a way to encode policy This results in Universal SFs, (s, a, w). on which we base our work.
The task encoding vector w can also be related directly to the rewards themselves via the underlying basis features (#):
R++1 = ((S++1)
3.3 Canonical Approach to Learning Successor Features and its Limitations
The canonical approach for learning the basis features and successor features v for each state St and action A, time step t, with respect to policy æ. are achieved by optimizing the following SF Temporal- -iffeeeeee-efeeeeeeeee loss:
Lo. [1((5++1) + (V+ w)) - (S,, Ar, w)ll
(4)
where action (S++1). The basis features & are typically defined as the normalized output of an encoder, which the SFs e learn from concurrently (see Figure for an example).
However, when the basis features, o, must be learned from high-dimensional, complex observations such pixels, optimizing Eq. may result in the basis features, é, converging a constant vector. This outcome occurs because it can minimize the loss, as noted by Machado et al. 2020). which we will also prove mathematically below.
3.4 Proof by Contradiction: Representation Collapse in Successor Features
Consider the basis features function Φ(-) and the Successor Features (-(), omitting the inputs for clarity. The canonical SF-TD loss (Eq. is defined as:
1 Lev.v 2 (b( + Tv())) (-11
(5)
Using proof by contradiction, aim to show that when both x6-) and (/) constants across all states S, specifically when 6(-) = and (--)) with = 72),, the system satisfies the zero-loss conditions, leading representation collapse.
We start with the assumption that if d(-) = c1, v))) = c2, then L * Ve1 02 E R.
Substituting d(-) and 2-(-) into the loss function:
1
La.v =
2 |e + 702 - -22l2
(6)
is trivial to observe that e = (1 - 72, the expression for T.g.v as follows:
Lov
[111 >)ez + 702 =22
2
|1112
=
This contradicts our assumption that Lo.e for particular relationship between ei and c2-
Thus, we have shown that there exist constants Cı, 02 such that when d(-) = c₁ and (b-) C2 with c (1 7)c2, the system does satisfy the zero-loss conditions, resulting in degenerate solutions for Lo.u- ie. causing representation collapse. In this collapsed state, Φ(-) loses its ability to distinguish between different states effectively, causing the model to lose critical discriminative information and thus impairing its generalization capabilities.
Additionally, we also show empirically in Figure I (a-c) of the presence of representation collapse when learning using Eq. s this work, our method aims to mitigate these issues with a novel, simple approach for learning SFs directly from pixels.

---

Proposed Method
The key insight from the proof above (section 3.4 is that preventing representation collapse requires avoiding the scenario where the basis features @ become a constant vector for all states S, which would minimize the loss without contributing to meaningful learning. Below, will describe the steps taken our approach to mitigate these issues causing representation collapse.
We note that when the representations & form set of SFs, Eq. ① satisfied for some that also satisties Eq. Therefore, the approach we take to learn SFs is simply course of the learning v and w come to satisfy both of these equations, which can be achieved by using the following loss functions:
Features-
Successor
Flatten A
Network
Features
ensure that over the
(MLP layer-
Network (MLP) (266d layers)
norm) (256d. leyer)
(S 10)
2580 per action
L 2
Bu+1 - -((S++1) "w12" Tw (8)
O2 B4 RGB
Q-SF-TD Loss
Product a
Q w) scalar per action
encoding
Li 10 (S,, Ac, w) wwll
Encoder (shared)
256d, vector
(9)
Reward Prediction
where @( (S++1) is treated as constant Eq. Jusing stop-gradient operator, and 0 is the bootstrapped target:
Dat Product
it11
Loss
scalar
(( Basis
y R&+1 + max (SS++1, d', w) w
Features
Oc-1
(10)
L2-Normaiisation
: stop gradient operator
84 RGB
Flatten
Retl
Here, w is only altered by L. whereas SF sa and the basis features @ are learned via Ly.
Our proposed model for learning SFs. Starting from the top, the representations of state S₁ are learned us- ing the shared encoder, resulting in h2 The basis features (S++1) are the normalized output of the encoder using state S1+1. The task-encoding vector w is learned through the reward prediction loss (Eq. 8 Concatenated with w, the basis features and successor features are learned through computing the Q-values with and minimizing the Q-SF-TD loss function (Eq. ( A schematic for con- tinuous actions and previous approaches can be found in Appendix G and respectively.
Figure 2:
Specifically, our proposed approach can overcome representation collapse by treating the basis features ó as the L2 normalized output from the encoder of the SF 2/ network (Figure 2 because unlike in Eq. 4 Eq. g and Eq. ? are not minimized by setting
a constant
value, given that and Re+1 are not con- stants for all states S. Hence, there is nothing encouraging the network to con- verge to a constant vector, naturally avoiding representational collapse.
When the basis features are needed to learn the task encoding vector through the reward prediction loss (Eq. D we apply a stop-gradient operator treat the basis features fixed. As will demonstrate in section / "Analysis of Efficiency and Efficacy", this inclusion of stop-gra gradient operator crucial. Without learning both the basis features & and the task encoding vector w concurrently can lead to learning instability.
Next, will clarify how our approach relates to learning SFs, they are defined mathematically. Given the straightforward nature our approach, we refer to the SFs learned as "Simple SFs."
4.1 Bridging Simple SF's and Universal Successor Features
In Proposition I (Appendix we show that our approach ultimately produces true SFs, equivalent to the SFs learned using Eq. Proposition T does this by proving that minimizing our losses (Eq. 8 & 9 also minimizes the canonical SF loss used in Universal Successor Features (Eq. 4 Furthermore, Proposition| S supports the proof above (Section 3.4p that our approach minimizes these losses in a manner such that setting the basis features Φ to constant is not a solution. Once again, if
x) = C2 and C₁ = (1 7))2 then Eq. 8 Eq. are not minimized, due the fact that ý and
R++1 in Eq. 10 are not constants for all states S.
5

---

DQN
Simple SF (ours)
SF Random
SF + Reconstruction
SF + Orthogonality
APS (Pre-train)
(a) Center-Wall Egocentric
Center-Wall: Alloceutric
(c) Four Rooms: Egocentric
40000
35000
30000
30000
Cunulative
25000
25000
0000
20000
20080
15000
10000
10000
2000
5000
5000
Task
Tack
Task
Task
T
Taskk
Ta.lk
Task Task
Tasc
Tusk 2
Exposure
Exposune
Exposure 1
Expasune
Expesure 1
Expcoure 2
Figure 3: Continual Reinforcement Learning Evaluation with pixel observations in 2D Minigrid and 3D Four Rooms environment. Replay buffer resets at each task transitions to simulate drastic distribution shifts: Agents face two sequential tasks (Task & Task 2), each repeated twice (Exposure 1 & Exposure 2). (a-c): The total cumulative returns accumulated during training. Overall, our agent, Simple SF (orange), shows notable superiority and exhibited better transfer in later tasks over both DQN (blue) and agents with added constraints. Importantly, constraints like reconstruction and orthogonality on basis features can impede learning. The plots for moving average episode returns are available in Appendix for additional insights.
Learning Successor Features the Simple Way
The architecture for our network is shown in Figure which is broadly inspired by Liu and Abbeel [202T]. Pixel-level observations, St, are fed into a convolutional encoder that outputs a latent representation h(S,), which is used both to construct the basis features and the SFs. To construct the basis features, o( (S)), we simply normalize the latent representations h (via L2 normalization, following Machado et al. (2220). To calculate the representations (S,,, Ar, w), the latent rep- resentation is combined with the task encoding vector, w, and fed into a multilayer perceptron that generates one set of representations for each possible action, A,. These representations are then combined with the task encoding via a dot product operation to estimate the Q-value function, Q(Si, A,, w) = (S,, A,, w) w. The policy is then simply an e-greedy policy based on the Q-value function.
To learn the basis features (d) and representations (1)), minimize the losses in Eq. annd Eq. using minibatch samples of experience tuples (St, Ar, Re+ St+1, 1,4-1,000 w), collected while interacting with the environment and sampled from replay buffer which is similar to Mnih al. (015). Critically, only the task-encoding vector w is learned by optimizing Eq. g so a stop gradient operator applied the basis features ((.)) (see Figure 2 The successor features, sh. in the bootstrap target, g. actually come from a target network, J. which updated periodically by using the actual network. a common approach in deep RL [Mnih et al. (0115). The successor features eú, and all of the upstream
network parameters are learned by minimizing Eq.
9 The full algorithm used for training our
network is given in Algorithm 1 in Appendix B
Experimental results
The environments used in our studies are 10 x 10 2D grid worlds, 3D Four Rooms environments (Figure in Appendix D) and Mujoco. All studies were conducted exclusively using pixel observa- tions, as the primary motivation for this paper is to address representation collapse when learning with pixel observations.
The grid worlds offer both egocentric (partially) and allocentric (fully observable) scenarios while the 3D Four Rooms environments provide exclusively egocentric observations. The rationale behind selecting these environments was threefold: first, evaluate the agent's 's learning capabilities across varying levels of environmental visibility, second, to examine its ability to interpret spatial relation-

---

ships and distances, and third, to provide set of tasks where the transition dynamics are easy to quantify for constructing SRs that can serve as a comparison evaluate the SFs with.
For a more complex setting, we considered the Mujoco environment because it allows for direct manipulation of the reward function and domains switching, such moving from half-cheetah to walker, given that they both have the same action dimensions.
To evaluate the generalization capabilities of the learned SFs, our studies focus on continual learning setting. In the 2D grid worlds and 3D Four Rooms environment, agents are exposed to two cycles of two distinct tasks. These tasks involves changes in reward locations (as shown in Figure Dp & Figure 9d) and/or changes in environment dynamics (as shown in Figure a & Figure Pe). Additionally, we explored two different scenarios to better simulate real-world conditions. The first scenario involves resetting the replay buffer at each task transition, which emulates drastic distribution shifts typically encountered in real-world applications. The second scenario maintains the replay buffer across task transitions, allowing us to assess the agent's 's learning continuity in a more stable data setting.
In the Mujoco environment, agents exposed to one cycle of two distinct task as in this setting, primarily wish to evaluate how well the agents can adapt to new tasks and mitigating interference.
In all experiments, we make comparisons with several baselines, namely, a Double Deep Q-Network (DQN) agent [Van Hasselt et al., 2016 and agents learning SFs (2)) with constraints on their basis features (Ø). including reconstruction loss [Machado et al. [020]. orthogonal loss (Touati et al. 2022]. and unlearnable random features [Touati et al. 2022 Additionally, we compare with an agent that learns SFs through a non-task-engaged pre-fraining regime [Liu and Abbeel, 2021). The mathematical definitions of the constraints can be found Appendix [F] To ensure the robustness of our results, all experiments are conducted across 5 different random seeds.
6.1 2D Grid world
The 2D Gridworld environments were developed based on 2D Minigrid (hhealier-Boisse et al. 2023 We created two different layouts the 2D Gridworld environment, namely Center- Wall (Figure 9a) and Inverted-LWall (Figure (9)). In order to align the setting more closely with the canonical Gridworld environment as described by (Sutton and Barto [2018] we altered the reward function such that it returns reward of +1 when the agent successfully reaches the goal state and 0 otherwise. For the 2D Gridworld environments, the agents were trained for one million steps per task.
Figure Da presents the cumulative returns for the Center Wall environment with egocentric observa- tions, while Figure b shows the results for allocentric observations.
The results show that our agent learns as well as, if not better than, the baseline models. Furthermore, model, SF Simple, exhibited better
when analysing the cumulative total returns during training, transfer that the baseline models. Particularly, SFs that are Iearned with constraints on the basis features clearly struggle to Iearn effectively, either due to the additional computational overhead or because representations that fulfill those constraints do not lead to effective policy learning.
6.2 3D Four Rooms environment
We developed the 3D Four Rooms environments (Figure D using Miniworld [Chevalier-Boisvent et al. 2023]. In this environment, the state and action spaces are continuous. In the first task, the agent receives a reward of +1 when it reaches the green box and a reward of -1 when it reaches the yellow box and this alternates for the second task. The agents were trained for five million steps per task. Similarly, the results in Figure 3. show that our agent is able to learn effectively using egocentric pixel observations in 3D environment.
6.3 Mujoco
In order to demonstrates our model's capabilities with continuous actions, we consider the Mujoco environment. We followed the established protocol in Yarats et al. (202] for effective learning with pixels observations in this environment. We started in the half-cheetah domain, rewarding agents for running forward in Task For Task 2, we introduced scenarios with running backwards, running faster, and switching to the walker domain. The results are presented in Figure 4
7

---

DDPG
SF Ontingonality Simple SF (ours)
SF Random
SF Reconstruction
SF Orthogonality
TalCheetah Run Direction: Forward
Barkward
(b/Cheetah Run Speed: Normal Fast
(cCCheetah Walker (Run)
Moving Avg Norma lzed
Epiode Returns
0.00
64b
Steps
Steps
105
lask
lask
lask
lask
Exposure
Expesure
Exposure
Figure 4: Continual Reinforcement Learning results using pixel observations in Mujoco environment across 5 random seeds. Replay buffer resets each task transitions to simulate drastic distribution shifts. we started with the balf-cheetah domain in Task where agents were rewarded for running forward. We then introduced three different scenarios Task (a) agents were rewarded for rumning backwards, (b) running faster, and, in the most drastic change, (c) switching from the half-cheetah to the walker domain with a forward running task. To ensure comparability across these diverse scenarios, we normalized the returns, considering that each task has different maximum attainable returns per episode. We did not evaluate APS (Pre-train) here because it struggles in the Continual RL setting, even in simpler environments such as the 2D Minigrid and 3D Miniworld.
Across all scenarios, our model not only maintained high performance but consistently outperformed all baselines in both Task 1 and Task 2, highlighting its superior adaptability and effectiveness in complex environments. This contrasted sharply with other SF-related baseline models, which struggled adapt under these conditions.
7 Analysis of Efficacy and Efficiency
7.1 Comparison to Successor Representations
(al Decoded SR (Allocentric)
Can our SF-Icarning technique, like traditional SRs, effectively capture the transition dynamics of the environment (Stachenfeld et al. 2017]? To investigate, we first sought quantitative measure to compare SFs to SRs. To do this, we trained a simple non-linear decoder to assess which model's SFs can be most effectively de- code into SRs. We conducted this evaluation using both allocentric and egocentric observa- tions within the center-wall environment. The results, depicted in Figure S shows that our model demonstrate consistently high accuracy (lower errors) across both settings. This trasts sharply with SFs developed using recon- struction constraints or random basis features, which, while effective in egocentric settings, perform poorly in allocentric settings where feature sparsity is greater.
Mean Squared Error
7,020
7,024
7,011
0.084
0.557
Simple SF (oan)
SF Rundum
SF Recorstmation
APS (Ph-itmin)
(b) Decoded SR (Egocentric)
Squared Error
11.754
0.400
0.545
0.325
0.749
Simple
Recosstrction Orthogonolitt
Figure 5: Decoding performance comparison of models' SFs into SRs using a non-linear decoder in the Center-Wai environment. Ground truth SRs generated analytically using Eq. 21 described in Appendix N Lower Mean Squared Error values on the y-axis indicate better performance.
We next utilized 2D visualizations with geospa- tial color mapping to differentiate environmen-
tal locations, aiming to see if similar SFs that are proximate in neural space proximate in physical space. Using UMAP MMelnnes et al. 2018 for dimension reduction, our results (Figure 6 suggest that simple approach captures environmental statistics comparably to other models, but with less overhead for calculating the loss. Moreover, our technique consistently forms organized spatial clusters across partially, fully, and egocentric observational settings.
Additionally, we performed a correlation analysis in 2D Gridworld environments, comparing each spatial position and head direction against analytically computed SRs [Daya] 1993), further confirm-

---

Center-Wall
(Fully-Obs.)
(b)
Four Rooms
(Egocentric)
Geospatial
APS
SF
SF
SF
SF +
Simple SF (ours)
Color Mapping
+
Random
Recon-
Ortho- gonality
Q-TD +
struction
Reward
Figure 2D visualization of Successor Features in (a) the fully-observable Center-Wall environment and (b) the 3D Four Rooms environment. Each row represents different models' visualizations post-training, starting with geospatial color mapping of the layout in the first column, followed by comparisons of SF-based models. Clustering indicates the capture of environmental statistics. Despite this, well-clustered SF models, especially those with orthogonality constraints, may not always translate to effective policy learning, as seen in Figure In allocentric scenarios, SFs with reconstruction constraints struggle with minimal pixel variations, unlike in the distinct pixel changes in the Four Rooms environment. For more visualizations, see Appendix M
ing the robustness and adaptability of our model's SFs in various observational contexts (Table and Table in Appendix No.
7.2 Is Stop Gradient critical for learning?
Four Rooms
Previous methods that concurrently learn the ba- sis features, é, and the task-encoding vector w, often face challenges with learning efficiency and stability, particularly in environments char- acterized by sparse rewards. This issue is illus- trated in Figure 10 in Appendix D of Ma et al. 2020 where optimizing the reward prediction loss (Eq. 8 can inadvertently drive the basis features towards zero (ó + 0), causing signifi- cant representational collapse. Representational collapse not only reduces the discriminative ca- pabilities of á but also undermines the agent's ability to differentiate between distinct states, thus severely impacting the overall learning pro- cess.
1.00
With stop grad (ours)
Avg Episode Returns
Without stop grad
(Basis Rewards)
0.50
0.00
-025
1
Steps
1e6
Figure 7: Efficacy of the Stop Gradient Operator in the Four Rooms Environment. Agents without stop gradient operator exhibit degraded learning.
As depicted in Figure our solution involves the strategic use of stop gradient operator applied to the basis features Ø. This operator prevents the gradient from the reward prediction loss from updating basis features o, effectively decoupling the learning of & from w, thus ensuring that retains its critical discriminative statistics, allowing for effective learning as demonstrated in Figure /
7.3 Robustness to Stochasticity within the environment
How robust are the SFs learned using our approach as the environment dynamics become noisier? To explore this question and verify the robustness of our technique, we also created slippery variant of the top right and bottom left rooms, the
the Four Rooms environment (Figure ().. Specifically, agent experiences a "slippery" dynamic: chosen actions have certain probability of being replaced with random, alternative actions. This design mimics the effects of a low-friction or slippery surface, creating scenario where the agent's 's intended movements might lead to unpredictable outcomes.

---

DON
Simple SF lours)
SF Random
Reconstruution
Orthogonality
APS (Pre-train) SF Q-TD Reward
Four Rooms
(bi Four Rooms (Sippery Prob 0.15)
Ict Four Rooms (Sipoeryy Prob m 0.30
(d) Four Rooms (Siipery Prob m 0.451
00000
Total Return
60000
40000
40000
30039
Bic
Bak
bask
Back
Sak
(
(F)
(g)
Steps good policy (1e6)
Computational Speed
Training Duration
Frames Second
302
25
20
200
Hours
15
1000
Exposurc Task
Exposure
Exposure 2
Exposurr Task
Task
Task 1
Figure& 8: Efficiency analysis using 3D Slippery Four Rooms environment. (a-d): Robustness analysis to increasing levels of stochasticity. (e) Bar plot showing efficiency in learning, measured as steps to achieve policy that produces a reasonable level of performance, with low values indicating higher efficiency. (f) Bar plot showing frames per second achieved by the agent during gradient computation, back-pro nack-propagation, and interaction with the environment. These metrics provide insights into the computational efficiency and the real-time interaction capabilities of the agent across different tasks or conditions. (g) Bar plot showing the total training duration for completing two exposures of two tasks, demonstrating overall time efficiency. Collectively, these plots reveal that our agent not only learns tasks effectively but also excels in computational efficiency.
The results in Figure Ea-d demonstrate that our approach is robust to increasing levels of stochasicity. Notably, when the stochasicity is high (slippery probability >= 0.3), all other SF methods fail to leamn effectively in the second task, whereas our approach continues to perform well.
7.4 Efficiency analysis
How do alternative SF learning methods with extra loss functions, like orthogonality constraints, stack up against our approach in terms of efficiency? We analyzed the number of steps each method takes to learn an effective policy, using a performance threshold defined by the shortest expected episode length from the last 10 episodes. A shorter episode length indicates better performance, it signifies quicker goal achievement. We noted the timestep when each model first met or exceeded this threshold. Our results, shown in Figure S demonstrate that our method outperforms all baselines in learning efficiency. Additionally, our method leverages simpler compute blocks and loss functions, enhancing computational speed and reducing training duration, as shown by faster frame processing rates (Figure 8 and shorter overall training times (Figure 8g).. Therefore, our approach is more efficient than the baseline methods for learning SFs.
8 Discussion
In this work, we developed method for learning SFs from pixel-level observations without pre- training or complex auxiliary losses. By applying the mathematical principles of SFs, our system effectively learns during task engagement using standard losses based on typical training returns and rewards. This simplicity and efficiency are key advantages of our approach.
Our experiments demonstrate that our method learns SFs effectively under various conditions and surpasses baseline models in continual RL scenarios. It effectively captures environmental transition dynamics and correlates well with analytically computed Successor Representations (SRs), offering streamlined, efficient strategy for integrating SFs into RL models. Future work could build on this to create more sophisticated models that leverage SFs for enhanced flexibility in RL.
10

---

Limitations and Broader Impact
The algorithms we developed were evaluated predominantly in simulated environments, which may not capture the diverse complexity of real-world scenarios. A key assumption in our approach is that pixel observations are of good quality. This assumption is critical as poor image quality could substantially degrade the performance and applicability of our algorithms.
The use Successor Features in learning algorithmss as demonstrated in this work, offers sigmificant advantages, particularly in mitigating catastrophic interference. This capability is crucial for the development of machine learning systems that require continuous learning, such as in dynamic environments. For instance, autonomous vehicles operating in ever-changing conditions can retain learned knowledge while adapting to new information, enhancing their safety and reliability.
However, the enhanced capabilities of these systems also raise concerns. The ability of machine learning models to continuously adapt and learn can lead to challenges predictability and control, potentially making outcomes less transparent. As systems become more autonomous and capable of adapting over time, there's risk that errors or biases in the learning process could propagate more extensively before detection, especially if oversight does not keep pace with the rate of learning.
10
Acknowledgements
Raymond Chua was supported by the DeepMind Graduate Award and UNIQUE Excellence Scholar- ship (PhD). We extend our gratitude to the FRQNT Strategic Clusters Program (2020-RS4-255002- Centre UNIQUE Quebee Neuro-Al Research Center).
Arna Ghosh was supported by the Vanier Canada Graduate scholarship and Healthy Brains, Healthy Lives Doctoral Fellowship.
Blake A. Richards was also supported by NSERC (Discovery Grant RGPIN-2020- 05105, RGPIN- 2018-04821; Discovery Accelerator Supplement: RGPAS-2020-000 Arthur B. McDonald Fel- lowship: 566355-2022), Healthy Brains, Healthy Lives (New Investigator Award: 2b-NISU-8), and CIFAR (Canada AI Chair; Learning in Machine and Brains Fellowship).
This research was further enabled by computational resources provided by Calcul Québee and the Digital Research Alliance of Canada along with the computational resources support from NVIDIA Corporation.
We are grateful to Gheorghe Comanici, Pranshu Malviya, Xing Han Lu, Isabeau Prémont-Schwarz and the anonymous reviewers whose insightful comments and suggestions significantly enhanced the quality of this manuscript. Additionally, our discussions with members of the LiNC lab | Mila and early collaborators from Microsoft Research (MSR) have been invaluable in shaping this research. Special thanks to Ida Momennejad, Geoff Gordon and Mehdi Fatemi from MSR for their substantial insights and contributions during the initial phases this work.
References
Zaheer Abbas, Rosie Zhao, Joseph Modayil. Adam White, and Marlos C. Machado. Loss of plasticity in continual deep reinforcement learning. arXiv.org, 2023. doi: 10.488550/axxiv223337777755
Majid Abdolshah, Hung Le, T. G. Karimpanal, Sunil Gupta, Santu Rana, and S. Venkatesh. A new representation of successor features for transfer across dissimilar environments. International Conference on Machine Learning, 2021.
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. arXiv preprint urXiv:2008.03/56, 2020.
https://www.callulluueeeer ca/ httpp:///allianceean.. ca/en
https://mila.queett quebec/home

---

L. N. Alegre. A. Bazzan, and Bruno C. Da Silva. Optimistic linear support and successor features as a basis for optimal policy transfer. International Conference on Machine Learning. 2022. doi: 10,48550/arxiv22266. 11226.
Nishanth Anand and Doina Precup. Prediction and control in continual reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference Machine Learning, pages 501-510. PMLR, 2018.
André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup. Fast reinforcement learning with generalized policy updates. Pruceedings of the National Academy of Sciences, 117 (49):30079-30087, 2020.
Lukas Biewald. Experiment tracking with weights and biases, 2020, URL https: //w:: wandb com/ Software available from wandb.com.
Diana Borsa, André Barreto. John Quan, Daniel Mankowitz, Rémi Munos, Hado Van Hasselt, David Silver, and Tom Schaul. Universal successor features approximators. arXiv preprint arXiv:1112.07626, 2018.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne. and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http://githut www.thuu.commggoooo croom//oool//axx
Kianté Brantley, Soroush Mehri, and Geoff Gordon. Successor feature sets: Generalizing successor representations across policies. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11774-1178111 2021.
Wilka Carvalho, Angelos Filos, Richard L. Lewis, Honglak Lee, and Satinder Singh. Composing task knowledge with modular successor feature approximators. International Conference on Learning Representations, 2023a. doi: 10.488500/rxiv2330..20000 12305.
Wilka Carvalho, Andre Saraiva, Angelos Filos, Andrew Kyle Lampinen, Loic Matthey, Richard Lewis, Honglak Lee, Satinder Singh, Danilo J Rezende, and Daniel Zoran. Combining behaviors with the successor features keyboard. arXiv preprint arXiv:2310.1544 2023b.
Maxime Chevalier-Boiste Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid miniworld: Modular customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2300.118811 2023.
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 1993. doi: 10.1 10.1ne2/. 19w......9999.4... 1993.5.4.613.
David Emukpere, Xavier Mameda-Pineda. and Chris Reinke. Successor feature neural episodic control. arXiv.org, 2021.
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Marc G. Bellemare. Proto-value networks: Scaling representation learning with auxiliary tasks. International Conference Learning Representations, 2023. doi: 10.48550arxiv. 2304.12567.
Angelos Filos, Clare Lyle, Yarin Gal, Sergey Levine, Natasha Jaques, and Gregory Farquhar. Psiphi- learning: Reinforcement learning with demonstrations using successor features and inverse tempo- ral difference learning. null, 2021. doi: 10.48550kamiv22102.22666
12

---

Scott Fujimoto, D. Meger, and Doina Precup. A deep reinforcement learning approach to marginalized importance sampling with the successor representation. International Conference on Machine Learning, 2021.
Jonathan Godwin Thomas Keck*, Peter Battaglia, Victor Bapst, Thomas Kipf, Yujia Li, Kimberly Stachenfeld, Petar Veličković, and Alvaro Sanchez-Gonzalez, Jraph: A library for graph neural networks jax., 2020. URL http:://tttt/ii/ www/deeppmiddd/raap.pa.pa
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24 (A(12):0028-))) 1040, 2020.
Steven Hansen, Will Dabney, Andre Barreto, Tom Van Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint arXiv:/9060 1906.00030, 2019.
Tom Hennigan, Trevor Cai, Tamara Norman, Lena Martens, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020. URL http::///gtt com hom//deppmidddmm haauuu
D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science Engineering, 9(3): 90-95, 2007. doi: 10.1 M,11009CEEEE.00..5555
David Janz, Jiri Hron, Przemystaw Mazur, Katja Hofmann, José Miguel Hernández-Lobato, and Sebastian Tschiatschek. Successor uncertainties: exploration and uncertainty in temporal difference learning. Advances in Neural Information Processing Systems, 32, 2019.
Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Continual reinforcement learning with complex synapses. In International Conference on Machine Learning, pages 2497-2506 PMLR, 2018.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 75:1401-1476, 2022.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 141iv:1412.6980. 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Grabska-Barwinska.
Yehuda Koren. On spectral graph drawing. In International Computing and Combinatorics Confer- ence, pages 496-508. Springer, 2003.
Tejas D. Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J. Gershman. Deep successor reinforcement learning. arXiv: Machine Learning, 2016.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. arXiv preprint 10Xiv:21 2021.
Lucas Lehnert and Michael I. Littman. Successor features support model- and model-free reinforcement learning. CoRR abs/1901. /1437 2019.
Lucas Lehnert, Stefanie Tellex, and Michael Littman. Advantages and limitations of using successor features for transfer in reinforcement learning. arXiv preprint (iXXiv1708.00100 2017.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: /w0.//09999999 2015.
Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. International Conference Machine Learning, pages 6736-6747. PMLR, 2021.
13

---

Chen Ma, Dylan R Ashley, Junfeng Wen, and Yoshua Bengio. Universal successor features for transfer reinforcement learning. arXiv preprint XXiv:2001.04025,, 2020.
Marlos C Machado, Mare G Bellemare, and Michael Bowling. A laplacian framework for option discovery reinforcement learning. In International Conference on Machine Learning, pages 2295-2304 PMLR, 2017a.
Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. arXiv preprint arXiv: 1710. 1089, 2017b.
Marlos C Machado, Mare G Bellemare, and Michael Bowling, Count-basee exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 5125-5133, 2020.
Marlos C. Machado, André Barreto, and Doina Precup. Temporal abstraction in reinforcement learning with the successor representation. Journal of machine learning research, 2021.
Tamas Madarasz and Tim Behrens. Better transfer learning with inferred successor maps, Advances in neural information processing systems, 32, 2019.
Sridhar Mahadevan and Mauro Maggioni. Proto-value functions: A laplacian framework for learning representation and control in markov decision processes. Journal of Machine Learning Research, 8(10), 2007.
L. McInnes, J. Healy, and J. Melville. UMAP: Dimension Reduction. ArXiv e-prints, February 2018.
Uniform Manifold Approximation and Projection for
Leland Mclnnes. John Healy. Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018
Matt McLeod, Chun-Ping Lo, M. Schlegel, Andrew Jacobsen, Raksha Kumaraswamy, Martha White, and Adam White. Continual auxiliary task learning. Neural Information Processing Systems, 2022.
Volodymyr Mnih, Volodymyr Mnih, Koray Kavukcuoglu, Koray Kavukcuoglu, David Silver. David Silver, Andrei Rusu, Andrei A. Rusu, Joel Veness, Joel Veness, Mare G. Bellemare, Mare G. Bellemare, Alex Graves, Alex Graves, Martin Riedmiller, Martin Riedmiller, Andreas K. Fidjeland, Andreas K. Fidjeland, Georg Ostrovski, Georg Ostrovski, Stig Petersen, Stig Petersen, Charles Beattie, Charles Beattie, Amir Sadik, Amir Sadik, Ioannis Antonoglou, Ioannis Antonoglou, Ioannis Antonoglou, Helen King. Helen King, Dharshan Kumaran, Dharshan Kumaran, Daan Wierstra, Daan Wierstra, Shane Legg, Shane Legg, Demis Hassabis, and Demis Hassabis. Human- level control through deep reinforcement learning. Nature, 2015. doi: 10.1038//auure14236.
Mark W. Nemecek and R. Parr. Policy caches with successor features. International Conference on Machine Learning, 2021.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54-71, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
F. Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn; Machine learning in Python. Journal of Machine Learning Research, 12:2825 28330, 2011.
Sam Powers, Eliot Xing, Eric Kolve, Roozbeh Mottaghi, and Abhinav Gupta. Cora: Benchmarks, baselines, and metrics as platform for continual reinforcement learning agents. In Conference on Lifelong Learning Agents, pages 705-743, PMLR, 2022.
14

---

Chris Reinke and Xavier Stameda-Pinedie Successor feature representations. arXiv preprint arX1v:2110.15011, 2021.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, J. Kirkpatrick, K. Kavukcuoglu, Razvan Pascanu, and R. Hadsell. Progressive neural networks. arXiv.org. 2016.
Kimberly L. Stachenfeld, Matthew Botvinick, and Samuel Gershman. The hippocampus predictive map. Nature Neuroscience, 2017. doi: 10. 10.1038/nm.4650
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3: 9-44, 1988.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? arXiv preprint wwXi:22209.49955 2022.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference on artificial intelligence, 2016.
Guido Van Rossum and Fred L. Drake. Python 3 Referencc Manual. CreateSpace, Scotts Valley, CA, 2009. ISBN 1441412697.
Michael L. Waskom, seaborn: statistical data visualization. Journal of Open Source Software, 6(60);
3021, 2021. doi: 10.21105/joss.00221 URL httpt:ttt/soi...o org 10. 21105/joss. 03021
Omry Yadan. Hydra framework for elegantly configuring complex applications. Github, 2019.
URL //gp git.b.thhbb...... conm/fffcchooorreeeeecrhhhhaaa
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.
Jingwei Zhang, Jost Tobias Springenberg. Joschka Boedecker, and Wolfram Burgard. Deep rein- forcement learning with successor features for navigation across similar environments. In 2017 IEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2371-2372 IEEE, 2017.
15

---

A Appendix
This supplementary section provides detailed insights and additional information that supports the findings and methodology discussed in the main paper. Below is a brief overview of what each section contains:
Appendix Section
Appendix B: Pseudocode Implementation
Appendix C: Proofs and Theoretical Justifications
Appendix D: 2D Minigrid, 3D Four Rooms and Mujoco Environments
Appendix E: Experimental details
Appendix F: Agents
Appendix G: Our Architecture for Continuous Control
Appendix H: Models of Previous Approaches
Appendix I: Impact of learning rate variations on task encoding vector
Appendix J: Further Experimental Results
Appendix K: Experimental Results of SF + Q-TD + Reward vs SF Simple (ours)
Appendix Implementation details
Appendix M: Visualisations of the SFs in the 2D minigrid and 3D miniworld Environment
Appendix N: Correlation Analysis
B Pseudocode Implementation
Algorithm 1 Learning Simple Successor Features Online
1: Initialize task encoding vector
2: Initialize SF the network, SF the target network
3: for t= T, do
4: Receive observation S, from environment
5: A, + e-greedy using QS,,, w)
6: Send A, to receive S141 and R4+1 from environment
7:
a' argmax (e((S+11, b, w)TT Tw b
8:
ý = R4+1 + (S a', w)Tw
9:
Φ + L2 normalized output from the encoder of SF v network loss,, = (Sr, A,, w) T u 0)
loss (o w R1+1)²
12: Gradient descent on dhe and w
end for
C Proofs and Theoretical Justifications
In this section, we provide a proof sketch to show that minimizing the Q-SF-TD loss (Eq. 9 will also result in minimizing the canonical universal SF-TD loss [Borsa al. 2018 for learning the SFs (wb() E "R.".. For the sake of brevity, we consider tabular RI. setting, where state is the current state, s' is the next state, a is the current action, and r the reward of the transition tuple (s, a, s',r) and as per defined in the main text, w R is the task encoding vector and Φ(-) E IR" is the set of basis features.
Let LsF be the canonical universal SF-TD loss Borsa et al. 2018
1 2 ||((')) + rp((s' d', w)) (1s, a, w)l
LsF
(1I)
where a' arg Q(s', b, w)
arg max (bss' b) T Tw and is the discount factor. We treat b
(ss',, a', ur) as part of the bootstrapped target: ÜSF = ((8)) + ( a', w). which results semi- gradient methods (Sutton and Barto 2018). Subsequently, the gradient V for LsF (Eq. 4 is defined
16

---

as:
V Lsp - (6(6)
+ - (0, ( d', w) - a, w))
(12)
Next, previously discussed in section y the Q-SF-TD loss Lys which used to learn the successor features (+) is defined as:
L 2 || (s, a, w) w|l
(13)
where 1 r max (86,, , a', w)
Tu w is the bootstrapped target.
(Note: Eq. 13 and the bootstrapped target is the same as Eq. and Eq. I0 respectively, presented in
Section of the main text)
Following the same reasoning in Eq. the gradient V for Ly is defined as:
V V =- y15(8' ( ,a''w)))) w) un 1b(8, a, ur) w) un
(14)
Proposition 1 Optimizing Vo w V VyLSEW, where Lifp is the canonical loss for universal successor features (Borsa et al. 2018/.
Proof. Now, assuming that for any given state 8, the reward r for state s can linearly decomposed into the dot product of the basis features ((s) and the task encoding vector w, as suggested by Sutton
1988 Dayan 1993]. follows that there exists an optimal set of basis features (*(s). This optimal
set ensures that the reward can be accurately represented as the dot product of d² (-) and the task encoding vector w:
(s') w
(15)
where 8' the next state.
Thereafter, let us recall that the reward prediction loss Lw is defined as:
L
"(s')
(16)
(Note: This equation is the same Eq. presented in Section + of the main text.)
Substituting the assumption that we made in Eq. 15 into the reward prediction loss (Eq, 16
L 2 ((s')Tw)
(s') T w -6(s') ww)
(Subst. r = o" =s') w following Eq. 15
2
((0"(s') - <(s')) Tw)
(e(s')Tux) w)
(17)
Where e(s') is the difference between o² (s') and d(s'). Furthermore, Li O, then e(s') w = w c(s') 0.
Shifting our focus back to the gradient V Ln of our Q-SF-TD loss function (Eq. 14p.
V y Lap
77ss' al, w)
(888, a, w) w)
(Eq. 14
-w ( + rp(s' al, u - 1b( ur)
(Subst. r= (8') ur following Eq.
- w T - + 7 (ss''',' a', w) vss w))w
(Subst. o² (s') ((s') e(s') from Eq. 17
-w (-V + e(s')) w
(Subst. definition from Eq. 12
w VpLSFW w e w
- V. 2 T w
(Subst.w Te(o)) = 2 Trg from Eq.
17
~ w V. L.LS
(18)
17

---

In conclusion, this proof demonstrates that the gradients V Ly computed using our proposed Q- SF-TD loss function (Eq. 13 effectively project the gradients yo Lşp from the canonical universal SF-TD loss function (Eq. 12 onto the task encoding vector w. This indicates that our loss function maintains the essential characteristics of the canonical form while aligning closely with the specific direction of the task encoding vector w.
D Environments
aa) Center-Wall (b) Inverted-LWalls
(c) Fully-obs. (top) & Partially-obs. (btm)
ELIPPERY
SLIPPERY
(d) Four Rooms (3D) (e) Slippery Four Rooms (3D)
(f) Egocentric obs.
(g) Half-Cheetah
(h) Walker
Figure 9: Overview of 2D Minigrid, 3D Four Rooms environments and Mujoco used in our studies with changing dynamics and rewards. Both 2D Minigrid and 3D Four Rooms environments utilize discrete actions while Mujoco utilizes continuous actions. All studies in this paper done using only pixel observations.
18

---

E Experimental details
In this section, we provide more details about the environments used in our experiments.
E.1 2D Gridworld Environments.
The specific parameters defining the 2D Gridworld environments are detailed in Table !
Table 1: 2D Minigrid Environment Specific Parameters
PARAMETER
VALUE
GRID SIZE
10 10
OBSERVATION TYPE
FULLY-OBSERVABLE & ARRTIALLY-OBSERVABLE OBSERVABLE
FRAME STACKING
No
RGB OR GREYSCALING
RGB
NUM TRAINING FRAMES PER TASK
MILLION FRAMES
NUM TASK PER EXPOSURE
NUM FRAMES PER EPOCH PER TASK
10K
BATCH SIZE
256
DECAY
20K FRAMES
ACTION REPEAT
NO
ACTION DIMENSION
OBSERVATION SIZE
84 x 84 x 3
MAX FRAMES PER EPISODE
400
TASK LEARNING RATE
0.0001
E.1.1
Center- Wall environment
(a) Center Wall Task
(b) Center-Wall
(c) Geospatial Color
Task 2
Mapping Task 1
Figure 10: Center-War environment and Geospatial Color Mapping
In the Center- Wall environment, a vertical wall splits the area into two distinct regions. Task features passage from the left to the right side the bottom, with the goal state located in the top left corner Figure 4 In Task 2, the layout is modified: the passage is moved to the top, while the goal state is relocated the bottom right comerr Figure TCp). These changes are strategically implemented to evaluate the agents' ability to adapt to simultaneous alterations in both the environmental structure and the goal location. To aid in visual analysis,
geospatial color mapping initially developed for Task (Figure I00c). This mapping effectively illustrates the spatial positioning within the environment and is particularly useful in the 2D visualization of the Successor Features and DQN Representations, providing a clearer understanding of how agents interpret and navigate the modified
environment (Figures 2V and 30
E.1.2 Inverted-Lwalls environment
In the Inverted-Lwald environment, we placed two L-shaped walls within the environment, one on the left and the other on the right, creating unique layout. This design results single, central path acting as a bottleneck, which the agent must navigate to reach the goal states. Specifically, to access the goal state located on the left side the environment, the agent required to traverse this central path while facing north (Figure ITa.. Conversely, to reach the goal state situated on the right,
19

---

Inverted-LWalls (b) Inverted-LWall
Task
Task 2
Figure 1I: Inverted-Lwalls environment
the agent must navigate the same path but facing south (Figure ID). This layout ensures that the agent consistently encounters and must maneuver this bottleneck area, regardless of the goal state's location.
E.2
3D Miniworld Environments.
The actions in this environment consists of moving Forward and Backwards, turning Left and Right. The specific parameters defining the 3D Miniworld environments are detailed in Table 2
Table 2: 3D Miniworld Four Rooms Environment Specific Parameters
PARAMETER
VALUE
OBSERVATION TYPE
EGOCENTRIC
FRAME STACKING
No
RGB OR GREYSCALING
RGB
NUM TRAINING FRAMES PER TASK
5 MILLION FRAMES
NUM EXPOSURE
2
EXPOSURE NUM FRAMES PER EPOCH PER TASK
2
100k
BATCH SIZB
32
DECAY
1 MILLION FRAMES
ACTION REPEAT
NO
ACTION DIMENSION
4
OBSERVATION SIZE
84 x 84
MAX FRAMES PER EPISODE
4000
TASK LEARNING RATE
0.001
SL.PPPER
(0.15, 0.3, 0.45, 0.6)
E.2.1 Four Rooms environment
(a) Layout
(b) Geospatial
Color Mapping
Figure 12: Four Rooms (3D)
The Four Rooms environment consists of four identical square rooms arranged in 2x2 grid, with passages connecting rooms and allowing an agent to move between the rooms (Figure 12p). Fach room in our 3D environment is designed with unique textures, deliberate choice to reduce the complexity associated with localization ambiguities often encountered more uniform settings. This variation in textures aids the agent in distinguishing between rooms based solely visual cues,
20

---

thereby simulating more realistic navigation scenarios. This setup also allows us to observe how visual diversity impacts the agent's ability to infer its location and navigate to specific goals, providing insights into the interplay between environmental features and SFs learning in a 3D spatial context. Depending on the task, the agent receives a reward of either or when it reaches the yellow or green box. Similar to the Center- Wall environment, we also create geospatial color mapping for the 2D visualization of the Successor Features and DQN Representations (Figure 12 p)
E.2.2 Slippery Four Rooms environment
LIPPER
SUPPER
Figure 13: Slippery Four Rooms (3D) layout
In the slippery variant of the Four Rooms environment, our goal is to rigorously test the robustness of agents in learning SFs under challenging conditions. Specifically, in the top right and bottom left rooms of this setup, the agent experiences a 'slippery' dynamic: chosen actions have a certain probability of being replaced with random, alternative actions. This design mimics the effects of low-friction or slippery surface, creating a scenario where the agent's intended movements might lead to unpredictable outcomes. Such setup is instrumental in assessing the agent's 's adaptability and the robustness of SF learning in the face of environmental unpredictability. This variant not only challenges the agent to adapt to unexpected changes but also provides valuable insights into the flexibility and resilience of the SFs when navigating environments where control and predictability are compromised.
E.3 Mujoco
In this work, we only utilised pixels inputs from Mujoco since our focus is on learning SFs directly from pixel observations. For domains, we chose both walker and half-cheetah. We broadly follow the same setup as Yarats et al. [2021] and included their model as baseline, which we denote as "DDPG" in our results (Figure
The codebase from their model is provided in the Unsupervised Reinforcement Learning (URL)
Benchmark repository [Laskin al. 2021 which we further described in the APS Agent in section The specific parameters we used for training in the Mujoco environment are detailed in Table 3
Table 3: Mujoco Environment Specific Parameters
PARAMETER
VALUE
FRAME STACKING
YES
RGB OR GREYSCALING
RGB
NUM TRAINING FRAMES PER TASK
2 MILLION FRAMES
NUM EXPOSURE
1
NUM TASK PER EXPOSURE
2
ACTION REPEAT
2
BATCH SIZE
256
FEATURE DIM
128
HIDDEN DIM
1024
OBSERVATION SIZE
84 x 84
MAX FRAMES PER EPISODE
4000
SF DIM
64
TASK LEARNING RATE
0.00001
TASK UPDATE FREQUENCY
10
21

---

F Agents
In this section, we describe how we create our agent as well as the ones we used for comparisons. In addition, we provide the mathematical definitions of the constraints used on the basis features. For all agents, we swept the learning rates for both the SF network and the task encoding (specific for all SFs agents) using gridsearch. The values ranged from 1e-1 to 1e-6, and the process was repeated using 5 random seeds in both 2D Gridworld and 3D Four Rooms environments. The same was also applied to the Double DQN agent [Van Hasselt et al. (016)) and we took extra care to ensure that the architecture and number of parameters were as similar as possible to our model. Detailed hyperparameters for learning SFs and the task encoding for our agent are outlined in Tables f and
E1 APS Agent
In our study, we take inspiration from the neural network architecture from Liu and Abbeel [202] from the Unsupervised Reinforcement Learning (URL) Benchmark repository| (Laskin al. [02] which utilizes PyTorch [Paszke et al., 2019). This repository was chosen for its robust implementation and served as the foundation for all SF-variaatt agents, including ours. Within the URL Benchmark, the encoder follows the Deep Deterministic Policy Gradient (DDPG) network architecture [Lillicrap ct al. 2013]. Notably, there is a discrepancy in the network architecture hyperparameters between the APS paper [Liu and Abbeel, 2021]) and the URL Benchmark repository. Given the practical implications of these differences, our implementation aligns with the hyperparameters specified in the URL Benchmark.
In line with the URL Benchmark's methodology, we initially employed the least squares method to determine the optimal task encoding However, we observed that this analytical approach was excessively sensitive our experimental context, particularly due to its reliance on the mini-batch samples. This sensitivity was especially pronounced in environments with sparse rewards, like those our study, suggesting that the least squares method might less suited for such settings. This challenge was not present in the original APS framework (Liu and Abbeel 2021], which was structured around distinct pre-training and fine-tuning phases. In contrast, our research focuses exclusively on continuous online learning, introducing unique challenges and dynamics not addressed in the APS paper [Liu and Abbeel 2021
F.2 Reconstruction constraints
At each time step f, the basis features ((S)) are generated from the current state S; using an encoder. Together with the action A,, these features are fed into a reconstruction decoder to predict the next state S141. Both the encoder and decoder are optimized using the reconstruction loss:
Lyecon = ||St+1 - S1+1l1
(19)
where S441 is the ground truth of the next state. The same set of basis features ø is also utilized in optimizing the Reward Prediction Loss (Eq. g) and the Q-SF-TD Loss (Fq. 9
F.3 Orthogonality constraints
At each time step I, the basis features & are generated from the current state St using encoder. Besides being utilized to optimize the Reward Prediction Loss (Eq. [) and the Q-SF-TD Loss (Eq.
D. the basis features é are also optimized with the orthogonality loss Koren 2003 Maggioni 2007 Machado et al. 2017b al
Mahadevan and
Tort B(SDD (I..) o (8+1))1 + E
(o(s)T ||((s) 11
(+))
(20)
22

---

Table 4: Simple SF Hyperparameters
PARAMETER
VALUE
OPTIMIZER
ADAM
KINGMA BA
2014
DISCOUNT(y)
0.99
REPLAY BUFFER SIZE
100K
DOUBLE
YES
VAN HASSELT ET AL..
2016
TARGET NETWORK: UPDATE PERIOD
1000
TARGET SMOOTHING COEFFICIENT
0.01
MULTI-STEP RETURN LENGTH
10
MIN REPLAY SIZE FOR SAMPLING
5000
FRAMESTACKING
NO
REPLAY PERIOD EVERY
16
EXPLORATION
G-GREEDY
LEARNING RATE
0.0001
RESET BUFFER WHEN TASK SWITCHES
NO
ENCODER CHANNELS
[32, 32, 32, 32)
ENCODER KERNEL SIZE
3, 3,
ENCODER STRIDE
[2, 1,
ENCODER NON-LINEARITY
RELU
BASIS FEATURES
L2-NORMALIZE (OUTPUT ENCODER)
FEATURE Ó DIMENSION
256
FEATURES- - NETWORK HIDDEN UNITS
256
FEATURES-TASK NETWORK NORMALIZATION
LAYER-NOW TANH
FEATURES-TASS NETWORK NON-LINEARITY
SF & DIMENSION
256
SF & NETWORK HIDDEN UNITS
256, SF t DIM ACTION DIM) RELU
[256,
SF & NETWORK NON-LINEARITY
Table 5: Task w encoding Hyperparameters
PARAMETER
VALUE
TASK W DIMENSION
256 VIIRONEETTDEPENDENTT (SEE TABLE
TASK LEARNING RATE
1 OPTIMIZER
ADAM KINGMA BA 2014
where states and s' are two different states sampled from the replay buffer D. The first term encour- ages the basis features d(S,) and ((S+1) to be similar and the second term promotes orthogonality by ensuring that the basis features of the different states Φ(s) and d(s') are distinct and decorrelated. Following Touati et al. 2022 we set the regularization factor X
F4 Random constraints
In this agent, the basis features are constrained to be unlearnable random features, which are defined during initialization. The SFs vi are subsequently learned on top of these predefined basis features. To guarantee that the basis features @ remain unlearnable throughout the training process, stop gradient operator employed.
F.5 Learning SFs through integrating losses
This agent learns Successor Features using a complex learning strategy that integrates three distinct losses: the SF-TD loss (Eq. 4. the reward prediction loss (Eq. B and the Q-SF-TD loss (Eq. D This multifaceted approach, proposed by Janz et al. 2019 aims to ensure that the learnt SFs satisfy all desired constraints.
23

---

G Our Architecture for Continuous Control
Actor Network (MLP + layer- norm) layer)
Actor Loss
Mean (Truncated Normal Distribution
Sampling
Action A
action clm,, vector
Features: Network
Successor
Flatten he
Features Network (MLP) layens)
(MLP layer- norm) (256d. layor)
O A4, w)
256d per action
Q-SF-TD Loss
10 encoding
RGB
a
Q Ag, w)
Scalar per aciion
256d, vector
Encoder (shared)
Reward Prediction
fe+1 scalar
Loss
((S+;1) Basis Features
Oze1
L2 2--oomalisation 256d
stop gradient
Loss functions
Reward Prediction Loss:
I, |M ((811)Tu[(
Q-SF-TD Loss:
Typ - |7 1 yymr
(,5,,,
Actor Loss:
Lao = Q(8,, M))
Figure 14: Our model adapted for continuous action spaces, based on the Actor-Critic architecture commonly used in DDPG [Lillicrap al. 2015) and implemented following the URL benchmark [Laskin et al.] [022]. This design modifies our original architecture to accommodate continuous action environments, enabling the model to handle broader range of control tasks. The model incorporates linear decomposition of Successor Features y and the task encoding vector w to compute Q-value, following Eq | Following the DDPG implementation in URL benchmark, actions are sampled from a truncated normal distribution, and LayerNorms are applied to normalize inputs to unit distribution.
24

---

H Models of Previous Approaches
Auxillary Loss (Reconstruction)
3
x RGB
w Task encoding
A one-hot vector
Reconstruction Decoder
Reward
Prediction
256d, vector
Loss
Dot Product
ft+1
Auxillary Loss (Orthogonality)
scalar
Encoder
ht
Flatten Φ 256d
MLP (256d, 3 layers)
TD Loss
Q ))
scalar per action
Successor
SR-TD Loss
Features
Network
(S,, )
O2 84 84 RGB
(MLP) (256d, layers)
256d per action
Loss functions
Reconstruction Loss:
Lrecon = || S441 1
Orthogonality Loss:
Lip (S.) + (S++))l + A (d(s) ( (s)))² -100))
SR-TD Loss:
5:0
Lg = 110(S + (1 ((S-1)) (SS, a)l
TD Loss:
LTD || + 7 Q(S,,,, a))
Figure 15: In order prevent representation collapse in the basis features d, previous methods
learning SFs from pixel observations often relied on an additional loss, such as reconstructing the state of the next time step S.+1 after executing action A, [Mahaad et al. 2020). Recent approaches in learning SFs include encouraging orthogonal representations in the basis features [Touaii et al.) [022]. A stop gradient operator is also used to prevent the SFs from updating the basis features Ø when optimizing the SF-TD loss. (Kulkarni al. 2016
25

---

I Impact of Learning Rate Variations on Task Encoding Vector
1.0
Avg Episode Returns
16:1
1e-2
le-3
16-4
le-6
2
Steps
1c6
Tasi
Expesure
Exposure 2
(a) Center-wall (Partially- -bsiellble))e
Avg Episode Returms
1e:1
0.5
10-3
16-4
1e:5
1e-6
0.0
Steps
Taisk 1
Task 2
Task
Taikk
Expesure
Exposure 2
(b) Center-wall (Fully-Observable)
Avg Episode Returns
16.2
led
16-5
1
0.0
0.5
1.0 Steps
1.5
2.0
1e7
Taisk 1
Task 2
Task
Task
Exposure
3D Four Rooms
Figure 16: Comparison of learning rates for the task encoding vector in grid worlds and 3D Four Rooms environments. Generally, a lower learning rate is required for the task encoding vector, despite its of simple reward prediction loss (Eq. 8) compared to the SF network, which needs more steps to converge due to its involvement in capturing complex environmental dynamics.
26

---

Further Experimental Results
In this section, present expanded illustrations of the results initially introduced in the main paper. These larger visual figures provide a clearer and more detailed view to enhance the reader's understanding our findings. Additionally, we include additional supplementary experimental results that were not featured in the main paper due to space limitations.
J.1 Single task results for 2D Minigrid and 3D Four Room environment
DQN
Simple (ours)
Random
SF
Reconstruction
SF + Laplacian
APS (Pre-train)
Aug Episode Returns
1.0
0.5
0.0
0.0
0.2
0.2
Steps
le6
Steps
le6
Steps
Steps
1e6
Wall: Partially-obs
(b) Center- Fully-obs
(c)
Slippery
Figure 17: Performance of agents trained on single task in both 2D Minigrid and 3D Four Rooms environments across random seeds. The Y-axis represents the moving average of the average episode returns. Our model, Simple SF (orange), performs comparably DQN (blue), though learns two Smaetions-Succesee Features (SFs) and the task encoding vector-while DQN only learns single function, the Q-value.
J.2
Continual RL results for Inverted-LWalls environment
DQN
Simple SF (ours)
SF Random
SF + Reconstruction
SF Orthogonality
APS (Pre-train)
1.0
Jug Episode Returns
Aug Episode Returns
0.8
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0.0
2.0 Steps
3.5
Steps
1e6
Task
Task 2
Task
Task
Task
Task 2
Task
Task 2
Exposure
Exposure
Exposure
Exposure
(a) Inverted-LWalls: Partially-obeervabe
(b) Inverted-LWall:: Fully-observable
Figure 18: Evaluation in Continual Reinforcement Learning setting across random seeds, without replay buffer resets at each task transition in the Inverted L-Walls environment. Here, the goal location alternates between the left and right sides with each task change, while the environment dynamics remain constant. (a) In the partially-observabee scenario, our agent demonstrates faster re-learning ability for new tasks compared other agents. (b) In the -ully-observable scenario, while our agent shows performance comparable to the DQN agent, it is slightly outperformed by the agent employing SFs with orthogonality constraints on its basis features. Notably, despite the superior performance of this latter agent in later tasks during Exposure 2, it initially faces difficulties in developing an effective policy, attributed to the added complexity of adhering orthogonality constraints.
27

---

J.3
Continual RL results for Center-Wall environment
DQN
Simple SF (ours)
Random
SF Reconstruction
SF Orthogonality
APS (Pre-train)
1.0
1.0
Avg Episode Returts
Avg Episade Raturns
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.5
1.0
1.5
2.0 Steps
2.5
3.0
3.5
4.0
0.0
0.5
1.0
1.5
2.0 Steps
2.5
3.0
3.5
4.0
le6
1e6
Exposure
Exposure
Expesure
Exposure
Center-wall:
Dartaaly-nbseervahl
Center Wall: Fully-observable
Figure 19: Evaluation in a Continual Reinforcement Learning setting across random seeds, without replay buffer resets at each task transition in the Center- Wall environment. In this setup, both the goal location and environment dynamics change with each task switch. (a) In the partially- observable scenario, agent demonstrates performance comparable that of other agents. (b) In the fully-observable scenario, our agent outperformed all others, with the agent employing SFs with orthogonality constraints its basis features coming in close second. Notably, while this latter agent shows improved performance in later tasks of Exposure 2, initially encounters difficulties in developing effective policy, which be attributed to the added complexity of adhering orthogonality constraints.
28

---

J.4
Continual RL results for Four Rooms environment
DQN
Simple SF (ours)
Random
SF Reconstruction
SF Orthogonality
APS (Pre-train)
1.00
Avg Episode Returns
0.75
0.50
0.25
0.00
0.25
0.50
-0.75
-1.00
0.00
0.25
0.50
0.75
1.00 Steps
1.25
1.50
1.75
2,00
1e7
Task
Task
Task
Task
Exposure
Exposure
Figure 20: Evaluation in a Continual Reinforcement Learning setting across random seeds, without replay buffer resets at each task transition in the 3D Four Rooms environment.
29

---

J.5
Continual RL results for Slippery Four Rooms environment
DQN
Simple (ours)
SF Random
SF Reconstruction SF Orthogonality
APS (Pre-train)
Slippery: 0.15
Slippery: 0.3
1.00
1.00
Avg Episode Retums
0.75
Avg Episode Returns
0.75
0.50
0.50
MMW
0.25
0.25
0.00
Muma
0.25
0.25
-0.50
-0.50
-0.75
-0.75
-1.00
-1.00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75 2.00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00 1e/
107
Expesure
Exposure
Exposure
Exposure
Slippery: 0.45
Slippery: 0.6
1.00
1.00
Avg Episode Returns
0.75
Avg Episode Returns
0.50
0.25
0.25
Gaimpay
0.00
0.00
-0.25
0.50
-0.50
-0.75
-0.75
-1.00
0.00
0,25
0.50
0.75 1.00 Stops
1.25
1.50
1.75
2.00
0.00
0.25
0.50
0.75 1.00
1.25
1.50
1.75
2.00 le7
1e7
Steps
Task 1
Tase
Task 1
Taak
Task 1
Task
Task
Task
Exposure
1
Exposure 2
Exposure 1
Exposure
Figure 21: Evaluation in slippery Four-Rooms environment with varied slipperiness probabilities, without replay buffer resets at each task transition. This environment features slippery conditions the top-right and bottom-left rooms for both tasks, Task 1 and Task Both tasks have differing reward structures: In Task 1, rewards are set at +1 for the green box and -1 for the yellow box; in Task 2, this reward scheme is reversed (green box: -1, yellow box: +1). The diagram illustrates layout of the environment can be found in Figure Note: The APS Pre-trained agent was tested only slippery probability of 0.15; higher probabilities were not evaluated due to performance decline beyond Task of Exposure 1 when the slippery probability is 0.15.
30

---

DQN
Simple SF (ours) - SF + Random
SF
Reconstruction SF Orthogonality
APS (Pre-train)
Slippery: 0.15
Slippery: 0.3
1
1.
0 75
Episode Returns
0.75
0.50
0.50
Eaisods
25
0.25
0.550
0..00
-0.7
0075
1.00 Steps
1.50
1e7
Steps
Task
Task
Trsicc
Thak
Task
Task 2
Tasicc
Taat
Exposure
Exposure 2
Exposure
Exposure 2
Slippery: 0.45
Slippery:
1 00
1.00
Avg Episode Returns
0
Avg Episode Returns
0.75
0
0.25
0 00
0.00
0.25
0.50
-0 75
-0.75
-1
-1.00
0.00
0.25
0.50
0.75
1.00 Steps
1.25
1.50
1.75
2,00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2,00
1e7
Steps
1e7
Exposure
Figure 22: Evaluation in a Continual Reinforcement Learning setting across 5 random seeds in the Four-Rooms Environment with environmental changes, without replay buffer resets at each task transition. Task adheres to the canonical Four Rooms environment dynamics, while Task 2 employs the slippery variant, where chosen actions altered based the slippery probability to simulate environmental changes. Throughout both tasks, reward associations remain consistent: +1 for the green box and -1 for the yellow box. The layout of this environment depicted in Figure Note: The APS Pre-trained agent was tested only at a slippery probability of 0.15; higher probabilities were evaluated due performance decline beyond Task of Exposure when the slippery probability is 15.
31

---

J.6
Continual RL results for 2D Minigrid and 3D Four Rooms environment with Replay
resets
DQN
Simple (ours)
SF Random
Reconstruction
Laplacian
APS (Pre-train)
Moving Avg Episode Returss
1.0
1.0
1..0
0.00
0.50
1.00
1.50
2.00
Steps
Steps
Steps
Task
Task 2
Task
Task
Task
Task
Task 1
Task 2
Task 1
Task 2
Tasc
Task 2
Expasure
Expesure
Expesure
Exposure 2
Exposure 1
Exposure
(a) Center Wall: Pustially-obs.
(b) Center Wall: Fully-obs.
(c) Four Rooms
Figure 23: Continual Reinforcement Leaning Evaluation with pixel observations in 2D Minigrid and 3D Four Rooms enviroment. Replay buffer resets at each task transitions to simulate drastic distribution shifts: Agents face two sequential tasks (Task & Task 2), each repeated twice (Exposure 1 & Exposure 2). Moving average episode returns using most recent episodes in both egocentric and allocentric 2D Minigrid environments and egocentric 3D Four Rooms environment.
32

---

K
Experimental results of SF + Q-TD + Reward vs SF Simple (Ours)
In this section, we present the experimental results our agent (SF Simple) and the agent which optimizes the three losses (SF + Q-TD + Reward) simultaneously. For more information about this agent, see section F.5
K.1
Continual RL results for Inverted-LWalls environment
Simple (ours)
Reward
1.0
AM
M
1.0
4
Avg Epitode Returs
Avg Episode Returns
0.8
0.8
0.4
0.4
0.2
0.2
0.0
1.0
2.0 Steps
3.0
4.0
0.0
1.0
2.0 Steps
3.0
4.0
le6
1e6
Exposure
Exposure
Exposure
Exposure
Inverted Walls::
Partially observable
(h) Inverted I. Lallss: Fully- observale
Figure 24: Evaluation in a Continual Reinforcement Learning setting across 5 random seeds, with replay buffer resets at each task transition in the Inverted L- L-Walls environment. Here, the goal location alternates between the left and right sides with each task change, while the environment dynamics remain constant. (a & b) In both partially-observall agent, which optimizes three losses simultaneously (SF + Q-TD + Reward), experiences learning instabilities due to the higher complexity involved in managing all constraints.
and fully-observable scenario, the
33

---

K.2
Continual RL results for Center-Wall env environment
Simple SF (ours)
SF Q-TD Reward
LKM
1.0
2ug Episode Returns
Aug Episode Returns
0.8
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0.0
1.0
2.0
4.0
0.0
2.0 Steps
3.0
4.0 1e6
Steps
Task
Task
Task
Task
Task 1
Taskc 2
Task 1
Task 2
Exposure
Exposure
Exposure
Exposure
(a) Center-wall:
vuriaall-obbervvae
(b) Center-wall: Fully-abservable
Figure 25: Evaluation in Continual Reinforcement Learning setting across 5 random seeds, with replay buffer resets at each task transition in the Center- environment. In this setup, both the goal location and environment dynamics change with each task switch. (a & b) In both partially- observable and fully-observable scenario, the agent, which optimizes three losses simultancously (SF Q-TD + Reward), experiences learning instabilities due the higher complexity involved in managing all constraints.
34

---

K.3
Continual RL results for Four Rooms environment
Simple SF (ours)
Q-TD Reward
Avg Episode Returns
O
0.0
0.5
1.0 Steps
1.5
2.0 1e7
Task
Task
Task 1
Task 2
Exposure 1
Exposure
Figure 26: Evaluation in Continual Reinforcement Learning setting across 5 random seeds, with replay buffer resets at each task transition the 3D Four Rooms environment. Simultancous optimization of three losses (SF Q-TD Reward) slows the learning process, the agent requires more time to learn an effective policy.
35

---

K.4
Continual RL results for Slippery Four Rooms environment
Simple SF (ours)
SF + Q-TD Reward
Slippery: 0.15
Slippery: 0.3
1.00
1.00
Avg Episode Returns
Avg Episode Raturns
0.00
0.00
-1.001
-1.001
0.00
0.50
1.00 Steps
1.50
2,00
0.00
0.50
1.00 Steps
1.50
2,00
1e7
1e7
Expasure
Exposure
Exposure
Exposure
Slippery: 0.45
Slippery: 0.6
1.00
1.00
Avg Episode Returnts
Avg Episode Returns
0.00
-1.00
-1.00
Steps
Steps
Task
Task
Task
Task
Task
Task
Task 2
Exposure
Exposure
Exposure
Exposure
Figure 27: Evaluation in Continual Reinforcement Learning setting across 5 random seeds, with replay buffer resets at each task transition in the 3D Four Rooms environment. This environment features slippery conditions in the top-right and bottom-left rooms for both tasks, Task and Task 2. Both tasks have differing reward structures: In Task 1, rewards are set at +1 for the green box and for the yellow box; Task 2, this reward scheme is reversed (green box: -1, yellow box: +1). The diagram illustrates the layout of the environment can be found in Figure 9 As observed, simultaneous optimization of three losses (SF Q-TD++ Reward) significantly impedes the agent's ability to learn effectively in stochastic environment.
36

---

Simple SF (ours)
Q-TD Reward
Slippery: 0.15
Slippery: 0.3
1.00
1.00
Avg Episodo Returns
Aug Episade Returns
0.00
0.00
-1.00
-1.001
0.00
0.50
1.00 Steps
1.50
2,00
0.00
0.50
1.00
1.50
2,00
1e7
Steps
1e7
Exposure
Exposure
Exposure
Slippery: 0.45
Slippery: 0.6
1.00
1.00
Avg Episade Returns
Avg Episode Returns
0.00
-1.00
1.50
-1.00
1.00 Steps
Steps
1e7
1e7
Task
Task
Task 1
Task 2
Task 1
Task
Task
Task 2
Exposure
Exposure
Exposure
Expccure
Figure 28: Evaluation in a Continual Reinforcement Learning setting across 5 random seeds, with replay buffer resets at each task transition in the 3D Four Rooms environment. Task adheres to the canonical Four Rooms environment dynamics, while Task employs the slippery variant, where chosen actions are altered based the slippery probability simulate environmental changes. Throughout both tasks, reward associations remain consistent: +1 for the green box and for the yellow box. The layout of this environment is depicted in Figure 9 Once again, simultaneous optimization of three losses (SF + Q-TD + Reward) significantly impedes the agent's 's ability to learn effectively.
L Implementation Details
For our experimental setup, we utilized Python 3 Van Rossum and Drake 2009 programming language. The agent creation and computational components were developed using Jax
the primary
[Bradbury et 2018 Godwin al. 2020], while Haiku (Hennigan al. 2020 was employed for implementing the neural network components. For data visualization. we used Matplotlib Hunter, 2007) and Seaborn [Wakoom| 2021 to generate line plots. Additionally, we utilized Plotly B for creating the violin plots and heat maps used in our correlation analysis, as well the 2D visualizations of the SFs and DQN Representations. We utilized Scikit-learn [Pedregosa et al. 201
in our correlation analysis studies as well the open-source Uniform Manifold Approximation and Projection (UMAP) tool [McInnes et al. 2018) to generate the 2D embeddings the SFs. The configuration and management of our experiments were facilitated by Hydra [Yadan] 2019) and Weights & Biases [Biewald 2020). All experiments, particularly those in the continual learning setting, were conducted using Nvidia V100 GPUs and completed within maximum of one day. The code used in the study will be released in the near future, following an internal review process.
'Plotly Technologies Inc. Collaborative data science. Montréal, QC, 2015. https://plot.ly.
37

---

M Visualizations of Successor Features
Given that Successor Features (SFs) are action-dependent, and considering the space constraints in the main paper, our visualizations here are more comprehensive. In the main paper, we primarily showcased visualizations for the forward action due to these limitations. However, in this section, we expand our focus to include visual representations for variety of actions, providing a more holistic view of the SFs' behavior and their influence across different action scenarios. This expanded visualization not only enhances our understanding of the SFs' multidimensional nature but also offers deeper insights into the agent's decision decision-making process and its interaction with the environment.
38

---

M.1 Center-wall Environment (Fully-observable)
Forward (Post-train)
Tum Left (Post-train)
Turn Right (Post-train)
Model
Initialization
Pre-train APS
SF +
Reconstruction
SF
Orthogonality
SF Random Features
SF + Q-TD Reward
Simple SF (ours)
Figure 29: 2D Geospatial Color-Mapped Visualizations of initial and action betion-based Successor Features the Fully-Obserwwe various RI. agents, each panel representing different agent and action.The first column illustrates the initial state of successor features before training, using geospatial color mapping for clear visualization. Subsequent columns correspond to successor features developed for specific actions: Forward, Turn Left, and Turn Right, also visualized using geospatial color mapping. In this scenario, only the agent learning SFs with orthogonality constraints well well-clustered representations after training. It's crucial to recognize, however, that while clustered representations may suggest effective learning, they do not automatically equate successful policy development. These visualizations highlight the varied encoding strategies of agents response to full observability and different actions.
Center- Wall Environment. This figure displays the successor features of
agent (Simple SF) learned
39

---

M.2 Center-wall Environment (Partially-observable)
Forward (Post-train)
Turn Left (Post-train)
Turn Right (Post-train)
Model
Initialization
Pre-train APS
SF
Reconstruction
Orthogonality
Random Features
Q-TD Reward
Simple SF (ours)
Figure 30: 2D Geospatial Color-Mapped Visualizations initial and oction-based Successor Features in the Partially-Observale various RL agents, each panel representing different agent and action. The first column illustrates the initial state of successor features before training, using geospatial color mapping for clear visualization. Subsequent columns correspond to successor features developed for specific actions: Forward, Turn Left, and Turn Right, also visualized using geospatial color mapping. Some agents demonstrate well-clustered representations after training, which typically correlates with improved performance compared to agents with more dispersed noisy features. It's erucial to recognize, however, that while clustered, color-mapped representations may suggest effective learning, they do not automatically equate to successful policy development. These visualizations highlight the varied encoding strategies of agents in response to partial observability and different actions.
Center-Wall Environment. This figure displays the successor features of
40

---

M.3 Four Rooms Environment
Forward (Post-train)
Backwards (Post-train)
Tum Left (Pust-Irain)
Turn Right (Pust-tram)
Model
Initialization
Pre-train APS
SF
Reconstruction
SF
Orthogonalityy
SF
Random Features
SF Q-TD Reward
Simple (ours)
Figure 31: 2D Geospatial Color-Mapped Visualizations of initial and action-based Successor Features in the 3D Four Rooms Environment. Agents operate with solely egocentric observations. Each panel represents the successor features of a different RL agent and action. The first column, using red, green, blue, and yellow to distinguish the four rooms, shows the initial state of successor features pre- training. Subsequent columns depict features for specific actions: Move Forward, Move Backwards, Turn Left, and Turn Right. Except for SF learned using unlearnable random basis features, most agents exhibit well-clustered representations post-training. However, it's important to note that such clustered, color-mapped representations, while indicative of effective learning, do not necessarily translate into successful policy development.
41

---

N
Correlation Analysis
Considering that the SRs are not normally distributed (Stachenfeld al. 2017]. we conduct our correlation analysis in the Grid world environments (Figure 2 and b) using the Spearman's s rank correlation. The SRs were analytically computed using the transition matrix T where T(s' s.a) denotes the probability of transitioning from state s to state s' given an action ~ .(- s):
SR - -TT--
(21)
where 0 S 7 < 1 discount factor and / identity matrix. The same policy я was used to generate the transition matrix T and adjust the final correlations. These adjustments account for less frequently chosen actions and for positions and head directions less likely to be encountered by the agent, outlined in the main text. Statistics regarding positions and head directions were collected using policy и.
In the remaining part of this section, we provide additional detailed violin plots to depict the correlation dynamics both the Center-wall and Inverted-LWail environments, covering scenarios that are both partially-observable and fully-observable. These plots are segmented into different stages: before training, after training, and the differences post-training. This segmentation offers comprehensive view of the agents' learning progression over time. Specifically for the Inverted- LWalls environment, a table is included to provide a summary of mean and standard deviation statistics for these correlations, thus offering clear quantitative perspective of our findings. Additionally, present heatmaps that showcase the correlation at each spatial position in the environment for various SF agents. These heatmaps further enrich analysis by visually representing the spatial distribution of correlation values, highlighting how different agents adapt to the environment.
42

---

N.1 Center-wall Environment (Parttally-observable)
0.5
Correlation
0.4
0.3
0.2
0.1
Simple SF (ours)
SF
SF
APS
SF
(Pre-train)
Random Reconstruction Laplacian
Q-TD
Reward
(a) Before Training
0.6
0.5
Correlation
0.4
Simple (ours)
SF
APS (Pre-train)
Random Reconstruction Laplacian
Q-TD
Reward
(b) After Training
A Correlation after training
2bp oooog اoo
Wii
Simple (ours)
(Pre-train)
Random Reconstruction Laplacian
Q-TD
Reward
(c) Difference (Before vs Afler)
Figure 32: Correlation analysis between learned Successor Features and analytically computed Successor Representation for all positions the Center-Wall Wall Environment under the Partially- observable scenario.
43

---

N.2 Center-wall Environment (Fully-observable)
0.4
piteeee vi
Correlation
0.2
0.3
Simple (ours)
APS (Pre-train)
Random Reconstruction Laplacian
Q-TD
Reward
(a) Before Training
0.6
0.5
Correlation
0.4
0.3
0.2
0.1
0.1
Simple
(ours)
(Pre-train)
Random Reconstruction Laplacian
Reward
(b) After Training
A Correlation after training
0.8
0.6
0.4
0.2
-0.2
Simple SF (ours)
SF
SF
SF
APS (Pre-train)
SF
+
+
Random Reconstruction Laplacian
Q-TD
Reward
(c) Difference (Before vs After)
Figure 33: Correlation Analysis between Successor Features and Successor Representation for all positions in the Center-Wall Environment (Fully-observable).
44

---

N.3 Inverted-LWalls Environment (Partially-observable)
0.5
Correlation
0.4
0.2
0.1
0.1
Simple
SF
(ours)
(Pre-train)
Random Reconstruction Laplacian
Q-TD
Reward
(a) Before Training
0.6
0.5
Correlation
0.4
0.3
0.1
Simple
(ours)
(Pre-train)
Random Reconstruction Laplacian
Reward
(b) After Training
Á Correlation after training
0.6
0.2
-0.2
Simple SF (ours)
SF
SF
SF
APS (Pre-train)
SF
+
+
+
Random Reconstruction Laplacian
Q-TD
Reward
Difference (Before vs After)
Figure 34: Correlation Analysis between Successor Features and Successor Representation for all positions in the SwerteddLLWlll Grid Environment (Partialt--bbeevable))
45

---

N.4 Inverted-LWalls Environment (Fully-observable)
0.4
0.3
Correlation
0.2
0.1
-0.2
Simple
APS
(ours)
(Pre-train)
Random Reconstruction Laplacian
Q-TD
Reward
(a) Before Training
0.5
0.3
0.2
0.1
Simple
SF
APS
(ours)
+
(Pre-train)
Random Reconstruction Laplacian
Reward
(b) After Training
Correlation training
0.6
0.4
0.2
0.2
Simple SF (ours)
SF
APS (Pre-train)
+
+
Random Reconstruction Laplacian
Q-TD
Reward
(c) Difference (Before After)
Figure 35: Correlation Analysis between Successor Features and Successor Representation for all positions in the Inverted-LWalls-Gre Grid Environment (Fully-observable).
46

---

N.5 Summary Statisties of the Correlation Analysis
Table Correlation Analysis against analytically computed Successor Representation in the Center- Wall Environment with mean and standard deviation of the correlations. The data are categorized into three stages: before training, after training, and the observed differences post-training. The left column: Partially-observal
scenarios in which our agent shows the highest correlation and greatest improvement post-training. The right column: Fully-observable scenarios where our agent and the agent with orthogonality constraints on basis features exhibit high correlation and significant post-training improvement.
PARTIALLY-OBS.
FULLY-OBS.
MODEL
BEFORE
AFTER
DIFFERENCE
BEFORE
AFTER
DIFFERENCE
SF + RECONSTRUCTION
0.233 0.09
0.29 0.08
0.051 0.09
0.111 0.07
0.101 0.04
-0.01 0.06
SF + RANDOM FEATURES
0.15
0.29 0.10
0.11 0.03
0,06 0.04
-0.05 0,07
SF + ORTHOGONALITY
0.14
0.29#0.09
0.04
0.1
0.34 0.08
0.22 0.14
PRE- TREIN (APS)
0.2110.07
0.17
-0.04
0.04
0.03
0.18
0.13 0.04
SF + Q-TD REWARD
0.21+0008
0.34 0.08
0.13+ 0.10
0.10+
0.26 0.06
0.16 0.11
SIMPLE SF(OURS)
0.24:1.09
0.321 0.08
0.07 0.10 0.11 0.09
0.334 0.07
0.221 0.13
Table 7: Correlation Analysis against analytically computed Successor Representation in the Inverted- LWalls-Grid Environment with mean and standard deviation of the correlations. The data are categorized into three stages: before training, after training, and the observed differences post- training. Notably, our agent demonstrated the largest improvement correlation well as the highest resulting correlation after the training period in both Partially-observabee scenarios (left) and Fully-observable scenarios (right).
PARTIALLY-OBS
FULLY-OBS.
MODEL
BEFORE
AFTER
DIFFERENCE
BEFORE
AFTER
DIFFERENCE
SF + RECONSTRUCTION
0,26 #0.1
0.30 0.07
0.04 0.09 0.11
0.10 0.03
-0.01 = 0.07
SF + ORTHOGONALITY
0.25 - 0.09
0.28 0.07
0.03 0.12
0.11
0.20 0.06
0.13
SF + RANDOM FEATURES
0.1
0.01 0.08 0.11
0.09 0.04
-0.01 0.08
PRE-TRAIN (APS)
0.09
0.19
0+
0.05
0.04
0.04
SF + Q-TD + REWARD
0.22+00099
0.29+ 0.08
0.06 0.11 0.12
0.29
0.17+ 0.12
SIMPLE SF(OURS)
0.26 0.1
0.07
0.08 0.11 0.11
0.09
0.06
0.22 1 0.11
47

---

N.6 Heatmap Visualization of SF Correlation in the Center-Wall Environment perrially-Obbervaalee
WW
0.405
0.250
0.204
0.410
0.454
(a) Center wall environment
(b) Before Training
1.200
0.159
0.122
0.025
0.077
0.089
0.041
5.034
0.084
0.107
3.052
0.2
0.017
0.122
5.200
2.375
0.202
3.352
5.450
0.223
0:334
5.000
0.244
0.242
3.377
0.424
0.122
0.200
0.30
0.279
0.225
2.257
0:035
0.027
0.075
-0.246
6.507
0.222
0.206
2.274
0.ce6
2.055
0.280
6.445
0.741
0.723
8.373
0.200
0.708
6.257
3.305
0.040
0.086
0.813
0.063
-3,013
0.231
+0.617
0.136
+0.149
-0.4
0:112
2.145
8-25
6:106
3.159
0.125
0.003
6.134
40003
-2.114
-0.00
-0.33
0.064
-0.00
46.035
(c) After Training
(d) Difference (Before vs After)
Figure 36: Correlation Analysis between Simple Successor Features (our model) and Successor Representation in the Center- Wall Environment (Partially-observable)
48

---

0.100
6.252
0.229
0.276
0.231
0.193
0.155
0.230
0.241
0.25
0.154
0.147
0.259
0.130
0.199
8,279
0. 1191
0:373
0.195
0.315
0.301
0.139
0.307
01333
0.227
0.105
0.040
0.214
0.305
0.135
0.089
0.203
0 201
0.145
0.250
0.353
0.359
0.036
0.390
0,233
0.352
0.275
0.411
W
0.443
0:166
6:306
0.207
0.272
0.370
0.347
0.103
0.272
0.222
8:303
0.000
0.324
0.250
0.300
0.304
0.225
0.304
0.309
8:464
0.472
0.294
6.476
0.303
0.340
(a) Center-wall environment
(b) Before Training
0.200
0.212
8,220
0.225
0.141
0.173
0.145
4.004
0.029
0.052
4.013
-0.550
0.262
0.350
0.237
6.79%
0.055
0.061
0.155
0.011
0.007
0.007
-2.55% 2.050
0.252
3.340
0.317
282
6.274
0.102
0.150
0.139
0.002
0.184
5.052
3.350
8.340
0.345
0.225
0.103
0.123
0.024
5.050
5.504
0.300
0.305
8.243
0.351
0.131
0.176
+..079
8.127
0.31
0.394
8.276
0.30
5.423
0.179
0.325
0.377
0.557
0.552
0.252
0.102
5.429
0.175
0.294
0.332
0,273
0.175
0.025
0.022
5.278
0.254
0:233
5.227
0.202
0.231
0.223
0.276
0.105
-2.121
6.115
+0.011
+0313
5.253
0.557
0.403
2.422
0.597
0.434
0.252
0.435
0.225
0.203
0.302
0.054
0.112
0.044
0.025
0.084
0.003
0.022
0,07
0.330
(c) After Training
(d) Difference (Before vs After)
Figure 37: Correlation Analysis between Successor Features with orthogonality constraints (SF + Orthogonality) and Successor Representation in the Center-Wan Environment (Parrially-observable
49

---

0.225
0.262
0.229
0.204
6.17%
6.210
0.253
6.222
0.316
0.203
0.100
0.200
N
0.211
0.000
0.199
8:276
0. 166
0:226
0.167
0.353
0.109
8:257
0. 3244
0.220
0.127
0.200
0.204
0.005
0.035
0,072
0.348
0 2012
0.227
425
0.427
0.355
0.085
0.345
8,255
0.331
0.204
0.424
W
0.44
0.166
8:227
0.179
0.257
0.300
0.44
0.222
0.223
0.233
8:267
0.131
0.332
0.215
0.061
0.304
0.725
0.334
0.377
8:450
0.472
0.279
6.455
0.425
0.203
(a) Center-wall environment
(b) Before Training
0.231
0.200
8.275
0.002
0.133
0.152
0.005
0.017
0.013
0.032
0.030
5.132
0.250
0.022
0.522
0.252
0.215
0,053
0.073
0.034
0.16
0.099
+0.03
0.55
0.210
0.303
0.244
0.003
0.025
5.000
0.200
0.315
0.315
0.37
0.337
0.212
0.105
0.032
0,023
0.028
0.345
0.294
0.184
0.374
0.051
0.611
0.059
6.148
0.131
8.363
0.304
5.472
0.125
0.207
0.200
0.237
0.575
0.220
0.424
5.524
0.213
0.202
0.592
0,245
0.425
0.024
0.028
5.555
0.211
0:251
0.253
0.307
0.222
0.230
0,285
0.507
0.003
-3.07
-2.814
5,317
0,411
0.425
2.443
0.397
0.287
0.244
0.457
8,340
0.225
0.003
0.006
0.161
0.085
0,053
0.02
0.033
0.025
0,077
0.328
(c) After Training
(d) Difference (Before vs After)
Figure 38: Correlation Analysis between Successor Features with Random un-learnable constraints (SF Random) and Successor Representation in the Center- Wall Environment (Partialyyoobservable)
50

---

0.273
0.276
0.203
0.185
0.133
0.170
0,05
6.226
0.197
0.246
0.107
0.117
0.210
6.162
N
0.200
0.179
0.255
853
0.200
0.303
0.210
0.375
0.204
0.008
0.000
0.083
0.332
0 2000
0.235
004
0.421
0:236
0.087
0.143
0,302
0.899
0.275
0.25
W
0.428
8,259
8:040
0.363
0.245
0.252
0.205
0.197
0.253
0.187
8.282
0.145
6.223
0.243
0.366
0.303
0:725
0.324
0.302
5.449
0.482
0.420
6.305
0.104
0.200
(a) Center-wall environment
(b) Before Training
0.233
0.202
0,223
0.120
0.103
0.110
5.000
0.003
0.041
0.017
00005
0.233
0.324
0.192
0.105
0.125
0.147
0.172
0.011
0.033
0.023
5.001
0.236
0.307
8.307
0.002
0.794
0.211
6.770
0.566
0.077
0.047
6.116
0.004
0.20
0.337
8.350
3.307
0.128
4.057
0.113
0.027
0.002
0.052
0.351
8.362
0.37
166
-8.033
0.041
0.044
-0.00%
5.475
0.273
0.204
0.253
0.214
0.027
0.326
0.555
5.479
0.225
0:252
0.203
0.273
3.234
0,250
0.272
5.473
0.257
0.167
0.255
0.207
0.253
0.083
5.224
0.242
0.275
0,222
0.207
0.227
0.200
0,245
0.215
0.016
-0.873
0.073
0.037
0.000
5.312
0.309
0.065
5.307
0.404
8:459
0.387
0.373
0.174
0.301
0,062
0.118
0.02
0.025
0.045
0.024
0,033
0,025
0.530
0.100
(c) After Training
(d) Difference (Before vs After)
Figure 39: Correlation Analysis between Successor Features with reconstruction constraints (SF + Reconstruction) and Successor Representation in the Center-Wall Environment (Partially-observabbl
51

---

0.209
0:357
0.200
0.000
5.281
0.194
0.105
0.300
0.317
0.250
0.331
0.300
0.226
0.202
0,277
0.124
0.397
0.359
0 2:223
0.135
0.2
0.201
0,274
0.083
0.109
8,226
0
0.122
0.041
W
0.277
0.256
0.349
0.336
0.275
0.239
0.234
0.257
0.208
0.229
8.241
0.109
0.278
0,411
0.220
0.181
6.220
0.336
0.307
8.200
3.207
0.100
6.242
0.29u
0.044
(a) Center-wall environment
(b) Before Training
0.200
0.040
0.177
8,274
0.212
0.057
0.073
-.074
0,016
0.013
4.120
5,077
0.002
0.350
0.028
0.135
0.115
0.020
0.033
4.254
0.004
0.021
4.074
0.000
0.170
0.138
0.155
0.695
0,00
0.071
+0.55
6.408
00.01
+3.66
0.571
6.156
0.160
0.151
10000
0.170
4.071
5.533
0.027
0.018
0.083
0.018
0.304
0.147
0.134
0:135
0.073
40.051
5.122
0.124
0.347
0.132
0.144
0.:25
0.373
0.253
5.137
0.225
0.35
0.134
0.152
0.237
0.067
0.202
5.247
0.204
0.127
0.578
0,200
0.177
0:152
0.000
5.055
0.131
0.228
0.212
0.157
0.144
0.177
0.225
0.103
.0.130
-2047
-..64
-3.131
+0.000
5.163
0.307
0.151
5.170
0.230
8.213
0.008
0.171
8.166
0.163
0.00
0.033
0.155
6.13
0.127
0.03
0.030
0.003
0.094
0.339
(c) After Training
(d) Difference (Before vs After)
Figure 40: Correlation Analysis between APS Pre-train Successor Features Liu and Abbeel 2021 and Successor Representation in the Center-Wall Wall Environment (Partialy-obbervvalee)
52

---

N.7 Heatmap Visualization of SF Correlation in the Center-Wall Environment (Fully-Observable)
WW
0.006
00 0.079
0.126
(a) Center-wall environment
(b) Before Training
0.212
0.004
0.000
0.337
2.204
0.307
0.115
0.122
0.323
8.355
0.224
0.143
6.140
0.523
0.005
0.322
0.230
0.207
0.247
0.255
0.310
0.172
0.019
0.170
0.307
0.113
0.008
0.070
0.215
0.227
9.338
0.040
0.057
0.277
0.359
0.360
0.103
0.352
0.271
0.670
0.377
0.341
0.243
0.327
0.-20
8.401
0.241
6.252
(c) After Training
(d) Difference (Before vs After)
Figure 41: Correlation Analysis between Simple Successor Features (our model) and Successor Representation in the Center-Wai Wall Environment (Fully-observable)
53

---

0.005
0:254
0.004
0.000
03.009
0.100
0.112
0.009
0.30
N
0..00
0.015
0.122
0.154
0.054
0.057
0.083
0.09
0.375
0.354
0.093
0.353
0.063
0.356
0.123
0.139
0.320
0.021
0.093
0.33
W
10.179
0:375
0,323
0.004
0.044
0.254
-0,027
0.175
0.222
0.000
0.061
+0 024
0.389
0.17
0225
0.017
0.224
0.046
0.162
6.131
0.082
0.100
0.274
0.033
0.203
(a) Center-wall environment
(b) Before Training
0.250
0.238
0.511
0.375
0.147
0.174
0.217
0.325
2.2
0.072
6.102
0.124
2.350
0.300
0.372
0.500
0.207
0.245
0.325
0.300
232
0.142
0.131
0.377
3.346
8.1.0
0.343
6.646
0.20
0.149
0.716
3.44
0.179
0.241
0.300
8.330
3.307
0.278
0.200
0.177
0.226
0.074
0.331
0.115
8.326
0.354
0.176
0.256
0.000
5.529
0.214
0.282
0.250
0.527
0.232
0.315
0.523
5.525
0.277
0.286
5.222
01:43
0.572
0,346
0.334
5.503
0.295
0.425
0.233
0,320
0.277
0.232
0.225
5.426
0.201
0.226
0.270
0.434
0.390
0.207
0.275
0.272
0.090
0.032
8.373
0.334
0.117
0.050
5.150
0.367
0.447
2.438
0.403
8.422
0.341
0.456
0.112
0.173
0.372
0.173
0.912
0.275
0.315
0.250
0.142
0.552
5.229
0.051
(c) After Training
(d) Difference (Before vs After)
Figure 42: Correlation Analysis between Successor Features with orthogonality constraints (SF + Orthogonality) and Successor Representation in the Center- Wall Environment (Fully-observable)
54

---

0.204
0.355
0.225
0.001
0.023
0.107
0.220
0.04
0.003
+0. 013
0.194
0.207
0.001
0.140
0.009
0.013
0.329
0.032
0.119
0.257
10.005
0.122
0.000
W
00.84
8:355
8,323
0.333
0.002
0.231
0.142
0:352
0.004
0.124
0.009
0.200
0.325
0.195
0.32
0.001
0.162
0.13
0.122
6.13
0.772
0.006
0.089
0.11
0.140
(a) Center-wall environment
(b) Before Training
2.038
0.043
8.267
0.035
3:072
139
0.152
00.12
4.
10.121
2.051
0.003
0.012
0.047
0.123
0.001
0.030
0.013
2.054
03.55
0.110
6.007
0.055
0.023
.0.099
-0.075
-0.111
+0.817
-3.16
+0.088
0.0%
40.00
00001
3.844
0.017
8.005
0.003
0,100
0.246
0.063
0.023
0.002
0.100
5.148
0.053
0.113
0.045
0.075
0.002
0.115
-3.066
-6.000
+0.006
5,030
0.044
0.045
0.054
0.097
0.047
0.063
0.087
5.023
0.073
0.002
0.128
0.001
2.043
0.067
0.075
5.041
0.114
0.077
0.080
8.863
0.103
0.043
0.001
5,011
0.011
60.00
0.058
+0.019
3.034
0.115
0.000
0.145
00101
-0.70
4.000
-..09
-2.128
+0.091
0.009
.0.010
-.011
0.077
0.822
0.055
8.087
-3.003
0.073
8.061
0.093
0.091
0.172
0.224
6.027
0.088
0.055
0.075
0.538
8.00)
0.047
(c) After Training
(d) Difference (Before After)
Figure 43: Correlation Analysis between Successor Features with Random un-learnable constraints (SF Random) and Successor Representation the Center-Wall Wall Environment (Fully-observable)
55

---

0.204
0.225
6.001
0.021
0.250
0.107
0.220
0.103
+0. 013
0.411
0.194
0.207
0.021
0.087
0.045
0.092
0.059
0.300
0.142
0.339
0.013
0.329
0.032
0.119
0.257
10.005
0.122
0.000
W
00.04
0:355
8,323
0.333
0.002
0.231
0.142
0.152
0.148
0.124
8.000
0.205
0.386
0.193
0.126
0.001
0.162
0.131
0.122
6.13
0.172
0.006
0.089
0.11
0.140
(a) Center-wall environment
(b) Before Training
0.035
0.000
0.015
0.015
4,005
6.117
0.045
4.001
0.000
0.000
0.030
0.123
0.006
0.175
0.033
0.07
0.103
00.07
4.057
0.007
0.127
3.837
0.145
0.007
+0.003
40.01
+0.11
4.117
00.17
0.088
0.574
00.00
067
6.100
3.831
0.097
0.04
8.006
0.135
0.023
0.042
4.005
0.054
0.012
0.074
0.040
0.115
0.122
0.058
0.137
-3,016
6.006
+0.093
5,022
0.114
0.080
0.128
0.113
0.082
0.307
0.105
5.053
0.001
0.266
0.327
0.155
0.002
0.332
0.133
5:029
0.031
0.000
0.128
0.324
0.179
0.027
0.082
5,059
0.924
0.091
0.023
0.097
0.134
0.340
0.335
0.155
-0.078
0.360
-2,037
-0.877
0.819
5,000
0.070
0.009
0.008
0.14
0.142
-2,038
0.107
8.005
0.13
0.012
0.002
0.072
6,053
0,026
0.053
0.133
0.02
0.024
0.007
(c) After Training
(d) Difference (Before vs After)
Figure 44: Correlation Analysis between Successor Features with reconstruction constraints (SF + Reconstruction) and Successor Representation the Center-Wall Environment (Fully-observable)
56

---

0.275
02
0.200
-0.006
0.223
0.034
0.773
0.075
0.042
0.035
0.08
0.253
10.04
0.114
0.061
0.021
0.023
0307
8,329
0.007
0.112
0.027
W
0.032
0.006
8.375
00012
0,077
0.30
0.033
0.006
0.013
0.003
8.545
0.053
0.050
0.02
0.100
0.054
0.CGI
-5.22
000
8.315
0.053
-0
014
8.063
0.043
0.079
(a) Center-wall environment
(b) Before Training
0.022
0.202
0.000
0.134
0.124
0.000
0.157
0.150
6.152
0.100
0.040
0.105
0.170
0.124
0.131
0.125
0.125
0.137
0.145
0.112
6.170
0.178
0.221
20300
8.128
0.351
0.140
0.001
0.179
0.149
6.746
0.165
0.140
6.570
6.179
6.100
0:041
8.000
0.337
0.102
0.140
0.153
0.100
0.242
0.153
0.112
0.132
0.33
0.180
0.332
0.363
0.205
0.138
6.173
5.148
0.140
0.200
0.124
0.130
0.213
0.226
0.201
5,042
0.073
0.300
0.333
0.154
0.054
0.236
0.202
5.528
0.135
0.145
0.044
0.235
0.177
0.033
0.372
0.076
5.153
0.107
0.200
5.223
0.203
0.311
0.191
8.25
0.201
0.100
0.157
0.101
0.049
0.097
5.309
0.19
0.200
5.588
0.205
8.226
0.314
6217
8.174
0.20
0.154
0.139
0.225
0.257
0.200
0.173
9.225
0.154
0.134
0.341
(c) After Training
(d) Difference (Before vs After)
Figure 45: Correlation Analysis between APS Pre-train Successor Features Liu and Abbeel 2021 and Successor Representation in the Center-Wall Wall Environment (Fully-observable)
57

---

N.8 Heatmap Visualization of SF Correlation in the Inverted-LWalls Environment Startialy-OObservalle
0.209
6.210
0.332
0.209
0.232
0.311
0.307
0.220
0.233
0.450
(a) Inverted-LWalls LWalls environment
(b) Before Training
0.204
0.204
0.201
0.073
0.008
0.051
0.332
0.213
0.125
0.037
0.999
0.
0.951
0.079
0.075
8.855
0.066
0,202
0.153
0.127
0.273
0.022
0.037
0.080
0.004
0.313
0.44
0.007
0.279
0.173
0.004
0.015
0.285
0.201
0.137
0.054
0.000
0:312
0.349
0:32
0.312
0.261
2.354
0.340
8.391
8.367
0.255
-2045
-0.009 0.033
(c) After Training
(d) Difference (Before vs After)
Figure 46: Correlation Analysis between Simple Successor Features (our model) and Successor Representation in the Inverted-LWalll-Gin Environment (Partialy-oobsrrvable)).
58

---

0.273
0:244
0.205
0.204
0:441
0.332
01:500
9.277
0.337
0.271
0.280
8.200
0.277
0.101
0.204
0.043
0.325
0.237
0.248
0.197
0.245
0.335
0.131
-0,046
0.220
0.336
0.333
6.270
0.129
0.100
0.120
0.061
0.241
6.278
0.315
0.265
0.415
0.141
0.244
0.208
0.260
0.122
0:259
0.224
0.205
0.260
9.363
0.656
0.2b2
0.244
0.272
0.233
0.284
0.206
0.31a
0.246
0.327
0.-01
0.226
0.232
0.401
0.334
0.304
0.250
aa) Hnverted-LWalls environment
(b) Before Training
5:325
0.141
0.243
2.420
11:227
0.382
0.234
0.327
0.275
0.235
0.852
0..02
0.144
5,612
0.022
0.025
0,046
15.01
DO0
0.4
5.223
0.148
0.254
3.354
0.301
0.257
0.230
0.175
0.212
.0023
0.174
0.113
0.215
0.055
-3.033
5.356
0.273
0:221
3.384
0.445
0.374
0.20
8.233
0.31
00.156
0.2
5.306
0.70
3.306
0.444
0.379
000
0.141
0.312
5.378
3.306
0.476
0:301
0.761
5.307
0.159
0.201
8.210
0.243
0.128
0.000
0.003
0.467
0.303
0.793
0.193
-0.094
-0.2
5.27e
0:208
0.208
0.244
0:432
0.079
0.102
0.075
0.100
0.118
0.167
0.028
0.001
5.247
0.201
0.198
0.285
.0.04
030
<C,118
0.001
-0.056
0.116
0.003
-0.4
0.53
0:304
0.700
2.388
0.207
8.360
0.347
0.304
6.267
0.273
0.057
0.042
0.055
0,134
0.059
0/02
0.145
5.010
0.207
0.08
(c) After Training
Difference (Before After)
Figure 47: Correlation Analysis between Successor Features with orthogonality constraints (SF + Orthogonality) and Successor Representation in the Inverted-L Waw.s-alrii Environment (Partially- observable)
59

---

0.303
0.202
0.255
0,302
0:424
0.133
0:257
5.424
0.43%
0.229
02228
0:134
0.225
0.133
0,222
0.048
00:00
0.383
0.273
0.234
0.207
0.35c
0.202
+0,005
0.200
0.321
0.203
0.275
M WW
0.133
0.133
0.143
0.40
0.296
0.322
0.203
0.213
0.105
0.196
0.382
0:266
0.205
0.133
0.288
0:242
0.288
0.224
0:249
0.290
0.249
0.345
0.401
0.228
0.233
0.288
0.221
0.263
0.240
0.220
0.262
0.204
0.207
0.261
0.338
0.205
0.242
0.204
0.400
0.239
0.206
aa) Inverted-LWalls environment
(b) Before Training
5:126
0,423
0:472
2.422
01A12
0.361
0.372
0.416
0.407
01143
10-19M
0.091
01:81
0.755
+2,012
0.007
0.015
-0.047
-5.024
10:155
0.4
5.204
0.935
0.226
0.277
0.522
0,000
3.244
0.299
0.277
0.124
0.200
0.130
0.043
0.125
0.007
5.116
0.31
8.20
0.055
0.133
0.384
0.297
8.776
0.275
6.000
0.2
5.111
0.103
0.142
0.201
0.332
0,297
0.243
0.135
5,170
5.153
0.177
0.797
0.710
5.947
0.229
0,20
8.261
0.157
0.024
0.029
0.001
0.315
0.263
00005
0.049
-0.000
0.014
-0.2
5.204
0:255
0.243
0.177
0.208
0.522
0.045
0.002
0.020
5.000
0.182
0.023
0.000
0.007
5.240
0.112
0.208
.0.003
-0.000
00001
-0.02
0.114
0.816
-0.4
5.36
0.207
0:223
0.302
0.101
8:361
0.332
6.422
0.203
0.001
0.054
0.015
6,035
0,259
0.053
0.070
5.006
6,004
0.097
(c) After Training
Difference (Before After)
Figure 48: Correlation Analysis between Successor Features with Random un-learnable constraints (SF + Random) and Successor Representation the mvverte-LWWall -aid Environment (Partially- observable)
60

---

0.304
0:3770
0.251
0.376
0:469
0:185
0.252
5.425
0,4
0.300
0.223
0.277
0.048
0.201
8,276
0.014
0.322
0.349
0.289
0.235
0.260
0.350
0.000
+0.043
0.250
0.122
0.30
6.265
0.123
0.135
0.137
0.085
0.203
0.2/2
0.290
0.234
0.419
0.167
0.292
0,250
0,277
0.134
0.228
0.207
0:249
0.384
0:241
0.3t
0.345
9.372
03575
0.24
0.242
0.203
0.279
0.267
0.230
0.272
0.003
0.332
0.152
0.333
0.303
0.244
0.302
0.413
0.355
0.293
aa) Hnverted-LWally environment
(b) Before Training
5:250
0.451
D:445
2.483
0.517
0.224
3.422
0,422
0.405
0.335
0.05
0.091
01:197
0.107
0,248
0.037
0.085
-0.035
-5.006
6.015
0.4
5.422
0.275
0.222
0.333
0.340
0.025
0.277
0.317
0.277
0.028
0.160
0.243
0.000
0.079
0.128
0.528
5.114
0.30
0.20
3.3
0.205
0.312
0.276
0.217
0.241
0.054
-3.045
-0.66
0.2
5/166
0.160
3.304
0.225
0.333
0.260
8.210
0.345
5.178
0.384
0.706
0:374
0.721
0.303
0.738
0.273
8.261
0.351
0.145
0.015
0.019
0.325
0.221
0.272
0.134
0.141
0.812
0.055
-0.2
5.232
0.254
0.254
0.298
0.306
0.522
0.094
0.035
0.023
0.121
5.075
0.094
0.054
0.046
5.223
0.207
0.225
.0.0%
0,013
-0,013
0.004
-0.05
0.031
-0.007
-0.4
6.31
0.291
0.766
0.336
0.314
8.376
0.354
0.300
6.305
0.113
0.71
0.055
0.27
6,047
0,000
0.053
0.059
5.034
0.002
0.27
(c) After Training
Difference (Before After)
Figure 49: Correlation Analysis between Successor Features with reconstruction constraints (SF + Reconstruction) and Successor Representation in the Wallls Grid Environment (Partially- observable)
61

---

0.207
0.237
0:000
0.310
0.324
0.205
0.304
0.233
0.312
0.322
0.252
-0.004
N
0.140
0.332
0.004
0.342
0:367
0.000
0.109
0.232
0.076
0.042
0.100
3.205
0.379
0.222
0.259
0.3394
0.380
0.083
02200
8,205
0.172
0.006
0.337
0.231
0.222
0.250
0.002
0.102
0.131
0.210
0.249
0.12
0.273
0.205
8.282
0.254
0.345
0.149
0.229
0.152
(a) Inverted-LWall environment
(b) Before Training
0.225
0.279
0.000
0.050
0.012
-0.23
0.035
4.002
0.003
0.053
0.130
0.006
0.313
0.350
0.070
0.229
0.373
0.237
.0.000
0.154
0.005
4.027
4.005
0.113
0.158
4.337
0.070
0.109
0.048
0.140
8.131
0.143
100
+0.01
-0.04
0.054
0.161
0.051
+3.447
-0.055
6.009
6.157
0.150
0.267
8.121
0.013
0.04
0.013
0.021
0.250
0.057
0:124
0.345
161
0.005
0.154
0.143
0.128
0.235
0.379
0,222
0.114
0.285
0.223
0.231
0.227
5.155
0.235
0.206
5.222
0.201
0.320
0.215
0.272
0.022
0.000
0.033
0.119
0.333
5.241
0.223
0.185
0.332
0.087
0.133
0.223
013
-2,073
0.007
-00.3
0.066
0.000
5.333
0.131
0.29
5.322
0.123
0.164
0.17
0.174
8.46
0.1G3
0.009
0.002
0.025
6,033
6.327
0.033
0.114
0.025
6,020
0.036
(c) After Training
(d) Difference (Before vs After)
Figure 50: Correlation Analysis between APS Pre-train Successor Features Liu and Abbeel 2021 and Successor Representation in the Inverted-LWalls-GGe -rid Environment (Partially-observable)
62

---

N.9 Heatmap Visualization of SF Correlation in the Inverted-LWalls Environment Saartaaly-Obbservall
-0.012
-0.023
0.233
0.120
0.022
0.147
0.127
0.032
(a) Inverted-LWalls LWalls environment
(b) Before Training
0.252
0.213
0.122
0.000
0.008
0.422
0.327
13135
0.487
0.006
0.108
0.129
0.414
0.023
0.145
0.275
0.242
0.270
0,003
0.245
0.000
0.086
0.407
0.333
0.399
0.173
9.275
0.367
0.131
6.025
0,225
0.425
0.008
6.115
0.224
0.343
0:33
0.230
0.337
0.375
2.300
0.305
8.376
0.333
0.322
(c) After Training
(d) Difference (Before vs After)
Figure 51: Correlation Analysis between Simple Successor Features (our model) and Successor Representation in the Inverted-LWalls-Gin Environment (Fully-observable)
63

---

0.308
0:214
2.247
0.209
0.042
0.085
0.149
5:224
0.247
0.257
0.809
8.005
0.030
0.003
0.203
0.587
0.005
0.224
0.206
0.093
0.160
0.135
0.071
-0.13
0.127
0.159
0.175
0.160
0000
0.035
0.194
0.089
0.004
0.053
0,077
0.171
0.119
0.205
-0.133
0.137
0.011
0.083
0.304
0.050
0.02
0.309
0.018
0.011
0.238
034028
0.230
0.032
0.002
0.233
0.345
0.204
0.025
0.511
0.003
0.103
0.129
0.157
0.122
0.140
0.047
0.172
0.243
0.226
aa) Inverted-LWall environment
(b) Before Training
5:245
0.427
0.202
2.422
03:227
0.325
3.424
0.306
0,348
0.255
0.259
190
0.145
0.125
0.197
0:234
0.255
0.112
6.101
10.099
0.4
5.428
0.555
0.375
0.300
0.357
0.282
0.302
0.203
0.255
0.292
0.216
0.207
0.205
0.139
5,177
0.2M
0223
0.304
0.377
0.37
0.274
8.770
0.211
0.134
0.05%
6.079
0.2
5,17
0.072
3.355
0.791
3.390
0.271
8.776
0.307
5.400
3.310
0.140
0:313
0.385
3.312
0.178
0.324
8.261
0.334
0.207
0.225
5,000
0,200
0.440
0.542
0.047
-0.2
5.325
0.264
0.355
0.54
0.320
0.205
0.257
5.355
0.422
0.353
0.312
0.252
0.223
0.100
5.249
0.247
0.402
0.222
0.274
0.013
0.111
0.143
0.301
0.073
0.247
0.009
-0.4
5.12
0:223
0.388
0.253
0.211
0.355
0.234
6.323
0.274
0.167
0.134
0.204
0.184
0.085
0.085
0.094
0.008
(c) After Training
Difference (Before After)
Figure 52: Correlation Analysis between Successor Features with orthogonality constraints (SF + Orthogonality) and Successor Representation in the DnvertedL-Wall Grid Environment (Fully- observable)
64

---

0.133
0.375
0.025
0.133
0.288
0.045
0.071
0.315
0.237
D.247
0.051
0.310
0.000
0.000
6.289
0.073
0.176
0.246
0:276
0.461
0.207
0.224
0.073
+0.11
0.116
0.087
0.103
0.199
0.417
0.337
0.094
0.085
0.251
+0.001
0.003
W
0.417
0.275
+0.34
0.334
0.083
0/0/1
0.032
9.244
0.008
0.335
0.084
0.143
0.257
0.175
0.333
0.207
0.037
0.178
0.374
0.4G4
0.200
0.132
0.190
6.164
0.052
0.100
0.045
0.223
0.100
aa) Inverted-LWalls environment
(b) Before Training
5.008
0.115
0.020
0.123
0.003
0.000
0.085
0.303
0.321
0.073
H0,032
01001
+0.076
0.000
0.211
+0.031
9.055
10.16
0.4
5.079
0.013
0.004
0.028
0.073
0.275
0.022
0.097
0.115
-.000
4,021
...00
0.103
-0.46
4.007
0.150
6.116
0.130
0.040
5.133
8.000
0.014
0.000
8.007
0.113
+0.67
+0.873
0.2
6.170
0.063
5.154
0.105
0.007
0.090
0.046
0.049
2.056
5.134
0.065
0.090
160
5.676
6.166
0.118
0.120
0.161
0.014
4.00
0.004
0.082
0.000
0.207
0.125
0.112
03.14
0.815
0.009
-0.2
5.006
0.1.4
0.080
0.275
0.033
0.577
0.157
0.004
0.012
0.133
0.030
0,006
0.533
0.035
0.007
5.140
0.103
8:257
0.132
0.154
.0.00
0.005
-0.004
0.003
+0.07
0.111
-0.000
-0.4
5.071
0.169
0.158
0.145
8.540
0.124
0.44
0.147
0.094
0.042
0.042
6,028
6.609
0003
0.066
0.013
0.206
0.08
(c) After Training
Difference (Before After)
Figure 53: Correlation Analysis between Successor Features with Random un-learnable constraints (SF + Random) and Successor Representation in the Inverted- Walld Walls-Grid Environment (Fully- observable)
65

---

0.128
0.375
0.332
0.139
6.208
0.032
0.009
6.318
0.254
0.241
0.073
0.305
0.058
0.004
6.377
0.525
0.111
0.325
0.270
0.043
0.100
0.194
0.003
-0.039
0.123
0.584
0.098
0.212
0.005
0.224
0.114
+0.008
0.214
-0.050
0.007
0.167
0.133
0.201
00.078
0.320
0.132
0.035
6.002
0.080
5.225
0.000
0.333
0.235
0.110
0.225
0.188
0.206
0,209
0.042
0.311
0.139
0.101
0.115
0210
6.159
0.052
0.100
0.000
aa) Inverted-LWalls environment
(b) Before Training
5.058
0.145
0.306
0.037
0.127
0.206
0.132
0.131
0.320
0.154
-0,000
03025
+0.007
-2.050
0.211
0.007
0.059
-0,055
10125
0.4
5.120
0.039
0.004
0.024
0.101
0.279
0.123
0.003
0.185
-0.005
-..03
0.104
0.107
0.012
-3003
5.116
0.135
0.060
5.100
0.108
0.083
0.63
8.002
0.10
+6.25
0.2
5.112
0.142
5.131
0.051
0.086
0.104
0.008
0.113
2.891
5.150
0.072
0.995
0.123
5.811
8.176
0.157
8.150
0.121
4.055
0.000
0.044
0.054
0.041
0.338
0.284
0.125
0.000
-3.25
0.048
0.050
-0.2
5.103
0.052
0.091
0.123
0.02
0.041
0.132
0.054
0.042
0.125
0.013
0.007
0.200
0.021
0.004
5.146
0.02
0.278
0.173
-0,000
6.0M
+0.017
0.03
-0.05
0.002
-0.007
-0.4
5.055
0.095
0.143
0.130
0.005
0.100
0.085
0.163
0.135
0.084
0.075
0.027
6,088
6,082
0.03
0.025
0.028
0.066
0.03
(c) After Training
Difference (Before After)
Figure 54: Correlation Analysis between Successor Features with reconstruction constraints (SF + Reconstruction) and Successor Representation in the Inverted-LWWan Seardd-Wallll--iiiee Environment (Fully- observable)
66

---

0.007
0.043
0.053
0.002
0.042
0.013
0.082
N
0.035
0.019
0,037
0.032
0.039
0.042
0.00
0.055
0.065
0.037
0.001
0.027
0,821
0.003
0.332
0.043
D.046
0.053
0.099
0.285
0.044
0.024
0.082
0.009
0.059
6,540
3.055
0.000
0.078
0.069
0.032
0.072
0,072
0.073
8,277
0.003
-8 005
0.006
0.024
0.006
(a) Inverted-LWalls environment
(b) Before Training
0.240
0.213
0.214
8,200
0.232
0.002
0.335
0.178
0.153
0.155
0.177
0.244
0.102
0.202
5,200
5.200
0.384
0.223
0.204
0.233
0.219
0.153
0.151
0.173
0.102
0.213
0.003
0.213
5.101
0.335
0.30
0.174
0.704
0.173
8.179
0.149
0.216
0.139
0.130
0.111
0.124
0.064
0.097
6.578
6.060
0:17
0.311
0.372
8.201
0.242
0.148
0,123
0.155
0.157
0.546
0.030
0.309
0.139
0.352
0.120
0.224
175
0,204
0.145
0.201
0.225
8.228
0,202
5.215
0.133
0.348
0,212
0.177
0.320
0.208
0.203
0.204
0.214
5,253
0.179
0.114
0.334
0.154
0.303
0.242
6.009
0.089
0.101
5.301
0.23
0215
0.300
0.150
0.268
0.313
0.113
8.40
0.153
0.389
0.155
0.153
0.125
0,075
0.153
0.214
0.047
5.225
0.004
(c) After Training
(d) Difference (Before vs After)
Figure 55: Correlation Analysis between APS Pre-train Successor Features Liu and Abbeel 2021 and Successor Representation in the Inverted-LWalll-GGe Grid Environment (Fully-observable)
67

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the paper's 's contributions and scope by clearly outlining our main achievement: the development of an algorithm designed to overcome representation collapse in learning Successor Features from pixel observations efficiently. We provide comprehensive evidence of representation collapse and review various existing approaches to address this issue, highlighting their computational demands and limitations in continual learning settings. Our contributions are precisely these comparative analyses and the introduction of more efficient algorithm suitable for both single task and continual learning environments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have included section on the limitations and broader impact of our work in section D in the main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
68

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The assumptions and the complete proof are provided in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The details of our architecture are provided in Figure 2 The pseudocode of our algorithm can be found in Appendix B In addition, the hyperparameters used in our experiments can be found in Appendix We intend to release the codebase in the near future after the conclusion of an internal review.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
69

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We used opensource libraries to perform the experiments in this paper. Details of these software can be found in the Appendix L We intend to release the codebase in the near future after the conclusion an internal review.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (tttps::/niis.cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.cc// http:/pbblcgguudeseeeee wtwpiieeeeeoSSSemesssseeeeeeeeee ssioooolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The hyperparameters for our experiments are listed Appendix for the environment specifics and [ for the egeents-speifics..
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
70

---

Answer: [Yes]
Justification: All computational experiments are performed over 5 random seeds and all statistical plots include error bars, which represent the standard deviation of the data.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information regarding the compute resources in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The data are collected using computational simulation, which does involve any humans or animals. Most of our studies focus on navigational tasks, which are critical for roboties and self-driving cars.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
71

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: We included a section on the limitations and broader impact of our work in section A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper focuses studying reinforcement learning agents in compu- tational simulation, therefore we do not foresee such risks with regards to this research paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
72

---

Answer: [Yes]
Justification: We have cited the papers of the environments used, in the main paper well as in Appendix E and
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdsourcing research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
73

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsoureing research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
74


Reviews:
[{"review_id": "l1YfdDnMBL", "reviewer": "Reviewer_7tgV", "summary": "This work introduces a new algorithm for training successor features in deep reinforcement learning. This is achieved by optimizing two separate metrics. The first requires the model to predict the cumulative reward following a full trajectory and optimizes the successor features and the basis feature. The second optimized metric requires the model to predict the reward at the next step using the basis feature and task vector, but in this case only optimizes the task vector. Experiments are conducted in grid worlds and a 3D Four Rooms domain which demonstrates that the proposed algorithm learns more consistently than other SF baselines and also supports faster task switching. Finally, supporting experiments show that the proposed algorithm is faster algorithmically and in terms of wall clock time than SF alternatives and that it also results in more separable SFs which correlate better with successor representations.", "questions": "I have asked some questions in the my review above and would appreciate those be answered. I do not have any other questions at this time.\n\nIf my question on Proposition 1 is answers and makes sense to me, and the concerns in Quality address I would be likely to advocate for acceptance, with agreement that the clarity would also be improved and figures restructured.", "limitations": "The limitations are stated in their own section and a good of consideration given towards the broader impact of this work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "# Originality\nThe decoupling of training into separate separate equations is new an intuitive idea. The authors note the inspiration from Liu et. al. (2021) however this algorithm is used as a baseline and clearly out-performed empirically.  Thus, it is clear that the changes made are material and have an impact on model performance.\n\n# Clarity\nThe paper is well written and sections are structured appropriately. Notation is intuitive, consistent and aids understanding. Figure captions are detailed which also aids clarity.\n\n# Quality\nI particularly appreciate some of the additional experiments conducted in support of the algorithm, such as the correlation between the learned SFs and SRs. The core experiments appear sufficiently challenging to separate the proposed algorithm from the baselines, and the baselines which are used are appropriate to challenge the proposed algorithm, The results are interpreted fairly as all algorithms do struggle on at least one domain where the simple SFs do not and consistently perform well.\n\n# Significance\nI do think this work could lead to future work and provide a helpful step in improving SFs and making them more practical. The significance is aided by the originality and simplicity of the approach as it is likely to spur new ideas quickly as a result.", "weaknesses": "# Clarity\nThe figures in this work are laid out poorly and this significantly hinders the readability of the paper. At the least it would help if a reader does not have to look past unseen figures on their way to look at the one being referenced - such as when looking for Figure 6 which comes after Figure 5. Also having the architectures in Figure 1 far from where they are needed and completely out of context is jarring and unhelpful. These architecture diagrams are also very difficult to follow and there is not clear mapping from what is depicted for some of the pieces to any explanation in the text or caption. Most of what is depicted in Figure 1d is not in Section 3, and similarly Section 5 is not detailed enough for me to map onto Figure 1e. I think more detail could be added to the figure itself and to the caption here. Lastly, including the loss functions in the figure, especially ones which have not been explained in the text, like orthogonality loss, is confusing. If these losses are not necessary in the main text then I don't think it is necessary in the figure and so I would remove them. With respect to Proposition 1, it would be better if this was at least in the main text - some kind of proof sketch would be better - but in the interest of space I see why it was omitted. Once again I do then just point to this as a part which could do with more explanation of why this matters and intuition on what it is true. If I am correct, Proposition 1 is the reason why Equations 5 and 6 cannot be optimized with the representation collapse strategy?\n\nSecondly, the Preliminaries section could have more elaboration. Equation 2 in particular is presented without any discussion and $\\gamma$ is not introduced at all. A reader with experience on RL and SFs will be fine but less experienced readers will likely be alienated. It would be ideal if this section could set up the ideas to come, and this is done to a degree with it being noted that representation collapse can still optimize Equation 4. More of this insight would just be helpful. Similarly, I would appreciate more discussion on why representation collapse is now not able to optimize Equation 5 and 6. This is merely stated without reason on lines 113 to 115.\n\n# Quality\nI am not certain I agree with the assessment from Section 7.2. While it is a worth experiment, the result of simple SFs being more correlated just appears to be due to the fact that it has a more linear latent embedding. This can be seen in Figure 5. SF+Reconstruction has very separable and clear clusters but they are just not organised in a straight line. So correlations - a linear metric - will not work. It seems to me that it would be more appropriate to try decode SRs from the SFs using a simple but nonlinear model and report the final accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate the opportunity to clarify and enhance our manuscript based on your observations. Please let us know if there is further clarification we can provide.\n\n# 1. Layout of Figure 5 and Figure 6\nThank you for your feedback on the order of Figures 5 and 6 in our manuscript. We recognize that aligning figure placement with their respective sections will enhance the manuscript's readability and coherence.\n\nCurrently, Figure 6 is introduced in Section 7.1's “Efficiency Analysis,” and Figure 5 appears later in Section 7.2's “Comparison to Successor Representations.” To improve narrative flow, we propose to swap these sections. This rearrangement will position 'Comparison to Successor Representations' before 'Efficiency Analysis,' ensuring that the figures align more logically with their related discussions.\n\n# 2. Modifications of Figure 1\nThank you for your valuable feedback regarding Figure 1. Due to space constraints in this rebuttal response, we invite you to refer to our detailed response provided to Reviewer Tczf under the section 'Splitting Figure 1' and in the general response above.\n\n# 3. Motivations behind Proposition 1\nThank you for the suggestion. Proposition 1 aims to mathematically demonstrate that the gradients from optimizing the Q-SF-TD loss (Eq. 5) effectively project the gradients from the canonical SF-TD loss (Eq. 4) along the task-encoding vector w. This projection is crucial in Continual RL as it aligns the SFs with different tasks, enabling the agent to adapt more rapidly to varying tasks.\n\nHowever, your comment on Proposition 1 being the reason why representation collapse is being mitigated is incorrect. For clarity on why representation collapse can occur, we have included an additional proof sketch below.\n\n# 4. Proof Sketch for Representation Collapse in Basis Features \n\nConsider the basis features function $\\phi(\\cdot) \\in \\mathbb{R}^n$ and the SFs $\\psi(\\cdot) \\in \\mathbb{R}^n$, omitting the inputs for clarity. The canonical SF-TD loss (Eq. 4) is defined as:\n\n\\begin{align}\n    L_{\\phi, \\psi} = \\frac{1}{2} \\left \\| \\| \\phi(\\cdot) + \\gamma \\psi (\\cdot) - \\psi(\\cdot) \\right \\| \\|^2\n\\end{align}\n\nAssume both $\\phi(\\cdot)$ and $\\psi(\\cdot)$ are constants across all states $S$, such that $\\phi(\\cdot) = c_1$ and $\\psi(\\cdot) = c_2$. \nIf $c_1 = (1-\\gamma)c_2$, then: \n\n$L_{\\phi, \\psi} = \\frac{1}{2} \\left \\| \\| (1 - \\gamma)c_2 + \\gamma c_2 - c_2 \\right \\| \\|^2 = 0$\n\nThis scenario illustrates that if both the basis features and SFs become constants, particularly with $c_1 = (1-\\gamma)c_2$, the system will satisfy the zero-loss conditions, resulting in representation collapse. In this state, $\\phi(\\cdot)$ loses its ability to distinguish between different states effectively, causing the model to lose critical discriminative information and thus impair its generalization capabilities. \n\n# 5. Introduction to RL in Preliminaries\nThank you for the comment. We will add the following text to section 3 to aid readers who may not be familiar with RL. \n\nThe RL setting is formalized as a Markov Decision Process defined by a tuple $(S, A, p, r, \\gamma)$, where $\\mathcal{S}$ is the set of states, $\\mathcal{A}$ is the set of actions, $r: S \\rightarrow \\mathbb{R}$ is the reward function, $p: \\mathcal{S} \\times \\mathcal{A} \\rightarrow [0,1]$  is the transition probability function and $\\gamma \\in [0,1)$ is the discount factor which is being to used to balance the importance of immediate and future rewards. \n\nAt each time step $t$, the agent observes state $S_t \\in \\mathcal{S}$ and takes an action $A_t\\in \\mathcal{A}$ sampled from a policy $\\pi: \\mathcal{S} \\times \\mathcal{A} \\rightarrow [0,1]$, resulting in to a transition of next state $S_{t+1}$ with probability $p(S_{t+1} \\mid S_t, A_t)$ and the reward $R_{t+1}$.\n\n# 6. Linear Latent Embeddings and Correlation Analysis\nThank you for your comment. We welcome the chance to clarify our use of UMAP for embeddings (Figure 5) and Spearman's rank correlation for analysis (Table 1).\n\nFirst, we use UMAP [1], a non-linear dimension reduction technique, for its effectiveness in visualizing complex relationships within Successor Features (SFs) in 2D space. It's crucial to note that UMAP does not imply linearity; the spatial arrangement of clusters should not be interpreted as linear relationships among features.\n\nSecond, our correlation analysis employs Spearman's rank correlation coefficient [2], outlined in Appendix K. This method assesses monotonic, non-linear relationships, suitable for our data's characteristics. Contrary to any suggestions of linearity, Spearman's correlation is non-parametric and does not assume linear relationships.\n\nWe will clarify these points in the manuscript to eliminate any ambiguity about our methods and to underscore the appropriateness and robustness of our analysis.\n\n# 7. Decoding SRs from SFs\nThank you for suggesting we use a simple nonlinear model to decode SRs from SFs. We implemented a single-layer perceptron with ReLU activation, training it for 4000 iterations using a 0.001 learning rate and Adam optimizer to ensure convergence within the center-wall environment.\n\nIn **Figure 3 of the General Response**, we present the mean squared error (MSE) results for both fully-observable (allocentric) and partially-observable (egocentric) settings. Our model achieved notably low errors, outperforming baselines in both contexts, highlighting its robustness and the effectiveness of our successor features in varying observational settings. This consistency was not observed in baseline models, such as SF + Random (green) and SF + Reconstruction (red), which showed variable performance.\n\nThese results confirm the strength and reliability of our decoded successor representations across diverse settings. We will incorporate this analysis into the manuscript. \n\n[1]  Leland et al., 2018. Umap: Uniform manifold approximation and projection. \n\n[2] Zar et al., 2005. Spearman rank correlation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6byHH3hytO", "reviewer": "Reviewer_2gcv", "summary": "This work presents a model architecture to learn successor features in reinforcement learning. It consists of optimizing Eqs. (5) and (6), i.e., a loss for learning the features and a loss for learning the task specific weights. It claims to avoid representation collapse. Experiments are conducted in common 2D and 3D tasks to show that the proposed method can achieve better performance and higher sample efficiency.", "questions": "See the weakness above.", "limitations": "The authors adequately addressed the limitations", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- A simple method that is easy to understand, and the presentation is easy to follow\n- Reasonable performance in the experiments", "weaknesses": "- Even though the paper claims that using the reward to train the task weights $w$ is new, this approach has been discussed before (Ma et al., 2020). Specifically, it has shown that using the reward $r$ to train $w$ is inferior to using the $Q$ values (Appendix D, Ma et al., 2020). Of course, the two algorithms are not identical, but it remains unclear why using the reward to learn $w$ is the right choice in the current paper. It is necessary to discuss this prior work.\n- Several other design choices need further explanation. L130-139 presents multiple design choices for the model architecture without proper discussion. For example, why the L2 normalization and the layer normalization are required? Why do we need to stop gradient at those specific places? It would be better to have ablation studies to show the importance of these choices.\n- Experiment results require further analysis. There is barely any analysis or reasoning for the proposed method. Each subsection in Sec.6 ends with \"our method is better\" without properly addressing **why** it can achieve better performance. This question is not answered in Sec.7 either. Moreover, the improvement over existing methods is only marginal (see Figs.2&3). More importantly, The quantitive results in Table 1 only show marginal improvement over Orthogonality with a significant performance overlap.\n\nMinor comment: Eqs.(5)&(6) are for scalars so a norm is unnecessary.\n\nReference:\n\n- Ma, C., Ashley, D.R., Wen, J. and Bengio, Y., 2020. Universal successor features for transfer reinforcement learning. *arXiv preprint arXiv:2001.04025*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate the opportunity to clarify and enhance our manuscript based on your observations. Please let us know if there is further clarification we can provide.\n\n# 1. Important differences between the Universal SFs and our approach\nWhile both our study and [1] utilize reward prediction loss, our approach integrates this with additional losses to directly learn SFs from pixel observations—a significant departure from [1]. \n\n**Our model:**\n1. **Learns basis features directly from pixels**, unlike [1] which uses pre-determined basis features.\n\n2. **Does not assume prior knowledge of task specifics**, contrasting with [1] where this is required, making our approach more applicable in continually changing environments.\n\n3. **Integrated SFs directly into the Q-value function**, simplifying and streamlining the learning without the need for redundant losses as seen in [1].\n\n4. **Ensures SFs play a crucial role** in performance, contrary to [1] where SFs play a minimal role due to the low weighting of SF loss.\n\n# 2, Reward Integration & Stop-gradient operator \nThe key distinction in our approach lies in the application of the stop-gradient operator during the learning of the task-encoding vector w with reward prediction loss (Eq. 6 in our manuscript). Unlike in Universal SFs [1]  where the basis features and the task-encoding vector w are learned concurrently, our approach prevents the updates to the basis features during this learning phase using a stop-gradient operator. This difference is crucial as we further demonstrated in an ablation study, **'Basis-Rewards' (Figure 4 in GR)**, since concurrent learning has shown to degrade learning efficiency (Figure 10 in Appendix D of [1]).\n\n# 3. Insufficient analysis\nWe respectfully disagree with the assertion that our work contains “barely any analysis.” Our work goes beyond theoretical discussions of representation collapse by providing empirical evidence (Figures 1a-c) and a **clustering analysis** (Figure 1c) that validate our method's effectiveness. This is complemented by a **mathematical proof sketch in Appendix C**, which explicates the gradient projections in our model, enhancing its applicability in continual learning scenarios. Our comprehensive analysis also includes **computational overhead comparisons** (Figure 6) and ablation studies (**Figure 4 in GR**) that reinforce the efficiency and effectiveness of our approach. Additionally, we will include a proof sketch detailing conditions under which representation collapse can occur (see rebuttal to reviewer 7tgV).\n\n# 4. Improvements are marginal\nThank you for your observations regarding the experimental results, specifically highlighted in Figures 2 and 3. Your feedback aligns with concerns previously noted by Reviewer Tczf regarding the apparent modest gains when measured using average episode returns. \n\nWhile average episode returns offer quick performance insights, they don't capture the full benefits of our approach. Hence, we've also evaluated cumulative total returns across tasks, which better reflect the agent’s ability to quickly learn and maintain effective policies over time\n\nOur analysis included in **Figure 2 in GR**, which consistently demonstrates significant improvements from our model compared to the baselines across various environments. Specifically, in the complex 3D environment, our model demonstrated significant improvement in cumulative returns, especially when the agent re-encountered previous tasks, *highlighting its enhanced transfer capabilities and effectiveness* in continual learning scenarios.\n\n# 5. L2 normalization and layer-norm\nL2-normalization is applied to both the basis features and the task-encoding vector w as commonly done [2,5]. We also normalize w before it enters the Features-Task network. These normalisations are to ensure consistent scale across inputs, enhancing optimization, training stability and preventsing any single feature from disproportionately influencing the learning process due to scale differences.\n\nAdditionally, we use layer-normalization within the Features-Task network to address un-normalized outputs from the encoder, a practice well-established in deep reinforcement learning to improve model robustness by conditioning the gradients [3,4,6,7]. \n\n# 6. Marginal correlation improvement over orthogonality \nThank you for your observation regarding the correlation improvements of SFs learned using our model. It's true that models enforcing orthogonality on features might show high correlations with discrete one-hot SRs due to their structured nature.\n\nHowever, our empirical findings, presented in Figures 2 and 3 of the manuscript, highlight that despite possible high correlation, SFs learned with orthogonality constraints often suffer from significant learning deficiencies. This issue becomes even more evident in the challenging Mujoco environments, as detailed in **Figure 1 of GR**.\n\nFurthermore, maintaining orthogonality constraints demands considerably more computational resources (Figure 6). Thus, while improvements in correlation might seem modest, our model offers a more balanced approach in optimizing both performance and computational efficiency. \n\n# 7. Norm is unnecessary\nThank you for the comment and we will make the revision in the final version. \n\n[1] Ma, et al., 2020. Universal successor features for transfer reinforcement learning\n\n[2] Machado et al., 2020. Count-based exploration with the successor representation.\n\n[3] Yarats et al., 2021. Improving sample efficiency in model-free reinforcement learning from images.\n\n[4] Yarats et al., 2021. Mastering visual continuous control: Improved data-augmented reinforcement learning.\n\n[5] Liu, et al., 2021. Aps: Active pretraining with successor features.\n\n[6] Ball et al., 2023. Efficient online reinforcement learning with offline data.\n\n[7] Lyle et al., 2024. Normalization and effective learning rates in reinforcement learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mWgvStSqjg", "reviewer": "Reviewer_Tczf", "summary": "The paper proposes a simpler method to learn Successor Features that avoids representational collapse. For this, the authors decompose the loss function to learn the successor features and task encoding separately. This allows for keeping the basis features fixed while learning the successor features, thus avoiding representational collapse. The experiments involve a continual learning scenario where robustness to task changes is evaluated. The authors show that the method can better adapt to changing tasks.", "questions": "- Should the method not be tested on environments where DQN itself cannot adapt to the new tasks at all? I wonder if the simple approach still holds when the transition dynamics become more complicated or the observations are more noisy.\n- For the Minigrid environment: Do you learn the successor features from pixels or do you use the built-in symbolic state representations?\n- Since DQN is also robust for the showed environments, I wonder how reward sparsity affects the performance of the different algorithms?", "limitations": "The authors have adequately addressed limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The problem is well motivated and the approach offers a simple solution to representational collapse when learning deep successor features. \n- The authors provided a number of insightful ablations. Especially that reconstruction based SF methods have trouble learning a good representation for fully observed settings.\n- The writing is clear and the method is presented in an understandable manner.", "weaknesses": "- The tested environments seem to be perhaps too simple for comparison both from the representational and task difficulty perspective, since DQN also has very good relearning capabilities in these environments. Why is it that DQN is better than the successor feature counterparts for the continual learning setting? This seems counterintuitive to me since successor features should be more robust than pure DQN.\n\n- The presentation of the figures has issues. Some Figures are pixelated, i.e., not vector graphics. (e.g., Figure 4 or other environment Figures). Also I think Figure 1 could be split into 2 figures for better readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate the opportunity to clarify and enhance our manuscript based on your observations. Please let us know if there is further clarification we can provide.\n\n# 1. Is DQN better than the SFs in Continual RL setting?\nThank you for your observations regarding the experimental results in Figures 2 and 3. While average episode returns offer quick performance insights, they do not fully capture the long-term benefits of our model. Thus, we also analyzed cumulative total returns across all tasks, as shown in **Figure 2 of the general response (GR)**.\n\nThese results confirm that our model quickly learns and maintains effective policies, especially in complex 3D environments where tasks recur (**Figure 2c-d in GR**). Our model significantly outperformed the baseline in cumulative returns, demonstrating its robustness and superior transfer capabilities compared to DQN, which showed little to no transfer effects and needed to re-learn tasks.\n\nWe will include these results in our manuscript to more comprehensively demonstrate our model's effectiveness in continual learning settings.\n \n# 2. Figure 4 Quality \nThank you for your feedback on the graphics in our figures. While all our figures are created with vector-based graphics for high resolution and scalability, Figure 4 is an exception. It uses pixel-based graphics to **accurately reflect the native format of the RL environments** and the inputs our models process.\n\n# 3. Figure 1 \nWe appreciate the suggestion to split Figure 1 to enhance readability. Acknowledging the density of the current figure, we will implement several modifications:\n1. **Simplification**: We'll remove the loss functions from Figures 1d and 1e, with detailed descriptions retained in Appendix E and the main text, respectively. This will help focus attention on the structural content.\n\n2. **Reorganization**: Figure 1d will be moved to the Appendix as it primarily presents common approaches rather than our novel contributions, ensuring the main text remains focused on our work.\n\n3. **Relabeling and Relocation**: Figure 1e will be renamed as Figure 2 and relocated closer to Sections 4 and 5 where it is first mentioned, aligning it more closely with its textual references and enhancing narrative coherence.\n\n4. **Visual Guidance Enhancements**: We will replace terms like “Q-SF-TD loss” with “$L_\\psi$: Q-SF-TD loss” and introduce color-coded information to improve figure-text integration such as, “Pixel-level observations, $S_t$​, are processed by a convolutional encoder to produce a latent representation $h(S_t)$, which is used to construct the basis features (*indicated by a yellow box in Figure 2*) and the SFs (*indicated by a green box in Figure 2*).”\n\nWe hope these changes will streamline the presentation and ensure the figures more effectively complement the text. \n\n# 4. Complex and noise in environments\nThank you for your comment on our model's effectiveness in complex, noisy environments.\n\nFirstly, our model's resilience to noise was proven in the “3D Slippery Four Rooms environment” (Section 6.1.3), where agents faced altered actions in Task 2. The results (Figure 3) demonstrate our model's superior robustness to induced stochasticity compared to baselines.\n\nSecondly, **we expanded our evaluation during the rebuttal phase to include the Mujoco environments**, using pixel-based observations and accounting for *continuous action spaces*. Following the setup in [1], we tested in scenarios like running backwards, running faster, and a major switch from the half-cheetah to the walker domain in Task 2. The outcomes (**Figure 1 in GR**) show our model consistently outperforming baselines across all scenarios, thereby showcasing its adaptability and effectiveness in more complex settings.\nThese results affirm our model's advanced capability to robustly handle diverse and challenging environments, making it highly suitable for practical applications with complex dynamics and significant noise.\n\n# 5. Pixel or Symbolic States observations for SFs?\nThank you for your question regarding the input modalities for Successor Features. In our work, **we exclusively use pixel observations across all experiments**. This choice is intentional, addressing a significant challenge in the field—the direct learning of Successor Features from high-dimensional sensory inputs such as pixels, which, as noted in [2], have historically posed difficulties for conventional methods and remain underexplored in the Successor Features literature [3, 4]. \n\n# 6. Sparse Rewards\nThank you for your question regarding sparse rewards. Like other DQN-based methods, our approach may face challenges in environments with sparse rewards, a recognized issue with bootstrapped learning methods. While our method is tailored for continual reinforcement learning, it is not specifically designed to address sparse rewards.\n\nWe acknowledge the need for mechanisms to better manage sparse rewards. Recent findings suggest that reconstruction-based objectives do not always capture task-relevant features effectively in such settings [5]. Integrating techniques that generate intrinsic rewards could help by providing more frequent learning signals.\n\nHowever, exploring these techniques further is beyond the current scope of our work. Our primary focus remains on demonstrating the viability of our approach in typical continual learning environments, laying the groundwork for future research to more comprehensively tackle the challenges of sparse rewards.\n\n[1] Yarats et al., 2021. Mastering visual continuous control: Improved data-augmented reinforcement learning.\n\n[2] Machado, et al., 2020. Count-based exploration with the successor representation.\n\n[3] Ma, et al., 2001. Universal successor features for transfer reinforcement learning.\n\n[4] Touati et al., 2023. Does zero-shot reinforcement learning exist?\n\n[5] Balestriero., 2024. Learning by Reconstruction Produces Uninformative Features For Perception.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mP5G3JHMld", "reviewer": "Reviewer_nkgZ", "summary": "This paper presents a method for learning Successor Features (SFs) from pixel-level observations in reinforcement learning (RL) by combining a Temporal-Difference (TD) loss with a reward prediction loss. This approach simplifies the learning process, improves performance, and speeds up learning compared to existing methods.", "questions": "Have the authors tested the proposed method with other RL algorithms and in more complex environments?", "limitations": "In addition to the limitations mentioned by the authors in the paper, please refer to my comments in the Weaknesses section for further details.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed method is simple and easy to implement, which is a significant advantage in practical applications. The effectiveness of the approach is demonstrated in both simple 2D and 3D environments.", "weaknesses": "1. The writing quality of the paper could be improved. The Introduction section reads more like an extensive review of related work rather than setting the context for the proposed method.\n2. The first three subplots in Figure 1 are difficult to understand without detailed background information. It is recommended to move these figures to the experimental section.\n3. The statement in lines 61 to 62, \"without any of the drawbacks,\" seems too absolute and should be toned down.\n4. The experiments only demonstrate effectiveness in a few simple 2D and 3D environments. To further validate the proposed method, it is recommended to test in more complex environments, such as Atari games, similar to APS.\n5. The experiments only tested the method with DQN, leaving its effectiveness with other RL methods unknown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate the opportunity to clarify and enhance our manuscript based on your observations. Please let us know if there is further clarification we can provide.\n\n# 1. Balancing Context and Review of Related Work\nThank you for your feedback on the structure of our introduction. We will revise it to better balance the context setting with the review of related work, enhancing both readability and clarity. In response, we will incorporate the following to the introduction section of our manuscript:\n\nSuccessor Features (SFs) are crucial in continual RL for decoupling environmental dynamics from rewards. Yet, current SF implementations often face representation collapse when learning from pixels, due to reliance on predefined assumptions and extensive pre-training. Our approach addresses these limitations by integrating an innovative neural network architecture that enhances computational efficiency and scalability.\n\nWe have validated our method with experiments in 2D and 3D mazes and the Mujoco environment (added during the rebuttal phase), detailed in **Figure 1 of the General Response (GR)**. Our findings show enhanced learning efficiency and adaptability, proving our model's broad applicability in RL scenarios.\n\n# 2. Placement and Purpose of Representation Collapse Analysis Plots in Figure 1.\nThank you for your feedback on the placement of the representation collapse analysis plots in Figure 1. We positioned these plots early in the manuscript to establish the central motivation of our research. While representation collapse is a well-known issue in Machine Learning, its empirical analysis within the context of SFs is a novel aspect of our work, warranting prominent placement to set the stage for the discussions that follow.\n\nIntroducing these plots at the beginning ensures that readers immediately understand the significance of the challenge we are addressing. This approach supports a cohesive narrative by linking the theoretical motivations directly with our proposed solutions and experimental validations.\n\nMoving these plots to a later section, such as the experimental results, could disconnect them from their theoretical context and reduce their impact on framing the research problem.\n\n# 3. The Exclusion of Certain Drawbacks in Our Method\nWe appreciate the chance to clarify the use of “without any of the drawbacks” in our manuscript. To address your concern, we will amend the phrase to “without some of the drawbacks.”\n\n# 4. Further evaluation in complex environments\nThank you for the suggestion to use Atari benchmarks. While APS's Atari setup involves pre-training and fine-tuning within environments that do not vary in features and reward functions, it is less suitable for assessing continual learning capabilities.\n\nInstead, we opted for a comprehensive evaluation in **Mujoco, utilizing pixel-based observations** [6], which further *demonstrates our model's capabilities with continuous actions*. We started in the half-cheetah domain, rewarding agents for running forward in Task 1. For Task 2, we introduced scenarios with running backwards, running faster, and switching to the walker domain. These are detailed in **Figure 1 in the GR**.\n\nAcross all scenarios, our model not only maintained high performance but consistently outperformed all baselines in both Task 1 and Task 2, highlighting its superior adaptability and effectiveness in complex environments. This contrasted sharply with other SF-related baseline models, which struggled to adapt under these conditions.\n\n# 5. Exclusivity to DQN\nThank you for your comment on the scope of our experiments. There are two reasons why we chose the baselines that we did.\n\n**First, we did not only compare to DQN, of course, but to several other techniques for learning SFs.** We did this because the focus of our work is learning SFs. Therefore, we selected other techniques for learning SFs as our primary baselines, such as with reconstruction or orthogonality constraints. Thus, we are comparing our approach to several other techniques, namely, those that are most relevant for the question of learning SFs. \n\n**Second, we chose DQN as a non-SF baseline because of its direct relation to the mathematical definition of SFs and Q-values, a common practice in SFs literature** [1,2,4,5]. This choice helps clarify the specific contributions of our approach in the context of well-understood benchmarks like DQN and DDPG [7].\n\nMoreover, **our primary goal was to develop a straightforward method for learning SFs, not to conduct a comprehensive benchmark across various RL algorithms.** More complex algorithms do not always lead to better performance, especially in settings with pixel-based observations, as shown in comparisons within the Mujoco environment where *simpler algorithms like DDPG often outperform more complex ones like SAC* [8] (see Figure 9a in [6]).\n\nWhile we acknowledge the value of broadening our evaluations to include a wider array of RL algorithms, our focus was on demonstrating the efficacy of our SF learning approach. Exploring performance with additional algorithms remains an important future research direction to enhance the generalizability of our findings.\n\n[1] Machado et al., 2020. Count-based exploration with the successor representation.\n\n[2] Ma, et al., 2020. Universal successor features for transfer reinforcement learning.\n\n[3] Touati et al., 2023. Does zero-shot reinforcement learning exist?\n\n[4] Janz et al., 2019. Successor uncertainties: exploration and uncertainty in temporal difference learning.\n\n[5] Barreto et al., 2017. Successor features for transfer in reinforcement learning.\n\n[6] Yarats et al., 2021. Mastering visual continuous control: Improved data-augmented reinforcement learning.\n\n[7] Lillicrap et al., 2015. Continuous control with deep reinforcement learning.\n\n[8] Haarnoja et al. 2018. Soft actor-critic algorithms and applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers once again for their valuable feedback, which has guided clarifications and improvements that we will include in the final revision of our manuscript. **We have attached a set of figures in this Author Rebuttal, which we denote as General Response (GR)**, to address the main concerns from the reviewers. The concerns fall broadly under the following themes: 

# 1. Complexity of the environments

During the rebuttal phase, we further evaluated our model in more complex settings using the Mujoco environments with pixel-based observations. We consider this benchmark to show the potential of our model in continuous action spaces. 

Following the established protocol in [1], we started with the half-cheetah domain in Task 1 where agents were rewarded for running forward. We then introduced three different scenarios in Task 2: agents were rewarded for running backwards (Figure 1a in GR), running faster (Figure 1b in GR), and, in the most drastic change, switching from the half-cheetah to the walker domain (same num of actions) with a forward running task (Figure 1c in GR). **To ensure comparability across these diverse scenarios, we normalized the returns, considering that each task has different maximum attainable returns per episode.**

In all tested scenarios, our model consistently outperformed all baselines in Task 1 and particularly, Task 2, highlighting its superior adaptability and effectiveness in complex environments. This performance sharply contrasts with other SF-related baseline models, which struggled to adapt under similar conditions.

# 2. Marginal improvements

We initially used average episode returns to provide quick insights into short-term performance, but recognize that this metric may not fully capture the long-term benefits of our model. To address this, we also evaluated cumulative total returns across all tasks, which are illustrated in Figure 2 in GR. 

These results demonstrate that our model not only learns effective policies more rapidly but also sustains these improvements, particularly in complex 3D environments where tasks are re-encountered (Figure 2c-d in GR). 

Overall, our model showed significant improvement in cumulative returns over the baseline models, highlighting its robustness and ability to transfer learning effectively across tasks. This contrasts with DQN, which exhibited little to no transfer effects and required re-learning from scratch, as evidenced by its performance in these scenarios. 

# 3. Simple nonlinear decoder
Reviewer 7tgV recommended a simple non-linear decoder to assess which model’s SFs most effectively decode into Successor Representations (SRs). We conducted this evaluation using both allocentric (fully-observable) and egocentric (partially-observable) pixel observations within the center-wall environment. The results, depicted in Figure 3 in the GR, demonstrate consistently high accuracy across both settings. This contrasts sharply with SFs developed using reconstruction constraints or random basis features, which, while effective in egocentric settings, perform poorly in allocentric settings where feature sparsity is greater. This analysis highlights the robustness and versatility of our model's SFs in varied observational contexts.

# 4. Stop Gradient Operator
The comments from Reviewer 2gcv prompted us to conduct an additional ablation study to elucidate the effectiveness of the reward prediction loss (Eq. 6) in our approach, compared to prior work [2] that faced challenges with similar methods. A key differentiator in our model is the application of a stop gradient operator on the basis features during the learning process with reward prediction loss.
We designed this study to specifically assess whether the stop gradient operator is essential for successful learning using reward prediction loss. The findings, presented in Figure 4a in GR, conclusively show that omitting the stop gradient operator leads to significantly reduced learning efficiency and policy effectiveness. 

Additionally, visual analysis of the SFs in Figure 4b in GR further demonstrates that concurrently learned basis features and task-encoding vectors without a stop gradient operator result in SFs with poor discriminative capabilities, undermining effective policy learning. These results underscore the critical role of the stop gradient operator in maintaining the integrity and effectiveness of our learning process, confirming its necessity for achieving the robust performance we report.

# 5. Modifications to Figure 1
Lastly, there were additional concerns regarding the configuration and density of Figure 1. As previously detailed in individual rebuttal responses to Reviewer 7tgV and Tczf, and for broader awareness, we will implement the following modifications:
Simplification: We'll remove the loss functions from Figures 1d and 1e, with detailed descriptions retained in Appendix E and the main text, respectively. This will help focus attention on the structural content.

- **Reorganization**: Figure 1d will be moved to the Appendix as it primarily presents common approaches rather than our novel contributions, ensuring the main text remains focused on our work.

- **Relabeling and Relocation**: Figure 1e will be renamed as Figure 2 and relocated closer to Sections 4 and 5 where it is first mentioned, aligning it more closely with its textual references and enhancing narrative coherence.

- **Visual Guidance Enhancements**: We will replace terms like “Q-SF-TD loss” with “$L_\psi$: Q-SF-TD loss” and introduce color-coded symbols to improve figure-text integration. For example, pixel-level observations, $S_t$, will be described in the text with direct references to their visual representation in the newly labeled Figure 2.


[1] Yarats et al., 2021. Mastering visual continuous control: Improved data-augmented reinforcement learning.

[2] Ma, et al., 2020. Universal successor features for transfer reinforcement learning

Meta-review: Scores 5 5 4 6

This is a solid paper, that is right on the threshold.

Criticism included that the tested environments are too limited and that the paper failed to properly discuss previous work from Ma et al 2020.  The rebuttal added further ablations and experiments, which is good and helps reviewers to increase their scores.  However, reviewer 2gcv wasn't completely convinced about the rebuttal against the concern that a key part of the paper is not completely new, but has been done by Ma et al. 2020 already.  However, 2gcv didn't answer the final rebuttal comment.

Overall the paper is good and the rebuttal did address most concerns, but at the end didn't convince all reviewers in all points.  The resulting score is just at the boundary.  I really appreciate papers trying SIMPLE approaches, so I tend to accept the paper.
Decision: Accept (poster)