Successor Clusters: A Behavior Basis for Unsupervised Zero-Shot Reinforcement Learning
Authors: Louis Bagot
Lucas Nunes Alegre
Steven Latre
Kevin Mets
Bruno Castro da Silva
Venue: Accepted by TMLR ()

Abstract: In this work, we introduce Successor Clusters (SCs), a novel method for tackling unsupervised zero-shot reinforcement learning (RL) problems. The goal in this setting is to directly identify policies capable of optimizing any given reward functions without requiring further learning after an initial reward-free training phase. Existing state-of-the-art techniques leverage Successor Features (SFs)---functions capable of characterizing a policy's expected discounted sum of a set of $d$ reward features. Importantly, however, the performance of existing techniques depends critically on how well the reward features enable arbitrary reward functions of interest to be linearly approximated. We introduce a novel and principled approach for constructing reward features and prove that they allow for any Lipschitz reward functions to be approximated arbitrarily well. Furthermore, we mathematically derive upper bounds on the corresponding approximation errors. Our method constructs features by clustering the state space via a novel distance metric quantifying the minimal expected number of timesteps needed to transition between any state pairs. Building on these theoretical contributions, we introduce Successor Clusters (SCs), a variant of the successor features framework capable of predicting the time spent by a policy in different regions of the state space. We demonstrate that, after a pre-training phase, our method can approximate and maximize any new reward functions in a zero-shot manner. Importantly, we also formally show that as the number and quality of clusters increase, the set of policies induced by Successor Clusters converges to a set containing the optimal policy for any new task. Moreover, we show that our technique naturally produces interpretable features, enabling applications such as visualizing the sequence of state regions an agent is likely to visit while solving a task. Finally, we empirically demonstrate that our method outperforms state-of-the-art SF-based competitors in challenging continuous control benchmarks, achieving superior zero-shot performance and lower reward approximation error.

Published in Transactions on Machine Learning Research (07/2025)
Successor Clusters: A Behavior Basis for Unsupervised Zero-Shot Reinforcement Learning
Louis Bagot
louis. louis. bagot.nao@gmail.ccmm com
IDLab Department of Computer Science, University of Antwerp imec
Lucas N. Alegre
Inalegreddinf.uuyys br
Institute of Informatics Universidade Federal do Rio Grande do Sul
Steven Latré
steven.latreeuuunnuurrrne be
IDLab Department of Computer Science, University of Antwerp imec
Kevin Mets
kevin. devin.metstn www.mmettttttteemmmmmm be IDLab Department of Electronics and Information and Communiication Tecknology, Faculty of Applied Engineering, University of Antwerp
smec
Bruno Castro da Silva
bsilba Scs. umass. edu
University of Mossachusells
Reviewed on OpenReview: opperevvee...... net/ formm? id= UB22Tt.3efF
Abstract
In this work, we introduce Successor Clusters (SCs), a novel method for lackling unsuper- vised zero-shot reinforcement learning (RL) problems. The goal in this setting is directly identify policies capable of optimizing any given reward functions without requiring fur- ther learning after initial reward-free training phase. Existing state-of-the-art techniques leverage Successor Features (SFs) funstinunctions capable of characterizing a policy's expected discounted sum of a set of d reward features. Importantly, however, the performance of existing techniques depends critically on how well the reward features enable arbitrary re- ward functions of interest to be linearly approximated. We introduce a novel and principled approach for constructing reward fcatures and prove that they allow for any Lipschitz ro- ward functions be approximated arbitrarily well. Furthermore, mathematically derive upper bounds the corresponding approximation errors. Our method constructs features by clustering the state space via a novel distance metric quantifying the minimal expected number of timesteps needed to transition between any state pairs. Building on these the- oretical contributions, we introduce Successor Clusters (SCs), a variant, of the successor features framework capable of predicting the time spent by policy in different regions of the state space. We demonstrate that, after a pro-training phase, our method can approxi- mate and maximize any new reward functions in a zero-shot manner. Importantly, we also formally show that as the number and quality of clusters increase, the set of policies induced by Successor Clusters converges to set containing the optimal policy for any task. Moreover, we show that our technique naturally produces interpretable features, enabling applications such as visualizing the sequence of state regions an agent is likely to visit while solving a task. Finally, we empirically demonstrate that our method outperforms state- of-the-art Sa-based competitors in challenging continuous control benchmarks, achieving superior zero-shot performance and lower reward approximation error.

---

Published in Transactions on Machine Learning Research (07 (07/202) (025)
S
S
Oml
Pre-training
Train (p(s)
Cluster
Train
(i(s) (seC,)
( a; w
Train Q(s, az
Distance in latent space
Features cluster occupancy
Navigate between clusters following preferences
reach goal states
min timesteps from to
X
Transfer
fit w
maximize reward
Transfer reward
average transfer reward per cluster
Zero-shot transfer. follow preferences w
The Successor Cluster framework. Pre-training: encodes a distance function based on the minimum number of time steps required to reach any state via goal-reaching RL. This achieved by learning a goal-conditioned action-value function, Q a: a;g). The induced distance function is used to identify used features, (s) 1c (s). The zero-shot agent learns Successor Clusters, viss, a; w), which model the expected discounted time spent visiting each cluster C, while solving task with reward function Tw ó.w. Transfer: Given any reward function the agent approximates it by fitting the reward weights: r r Φ w.rr Finally, the agent can follow the policy that maximizes the cluster preferences w, a zero-shot manner.
Figure 1:
The agent builds a latent space, spS), that
of clusters of the state space, {C)] that are then
1 Introduction
In Reinforcement Learning (Sutton Barto 2018 RL), an agent interacts with an environment, defined Markov decision process (MDP), performing actions maximize rewards in sequential task, based on its current state. However, for given environment, reward functions. For instance, single robot can perform very vast array of tasks, which might all be relevant in certain contexts. Classic RL algorithms are not equipped to deal with this, as they essentially overfit to a single reward function and would need re-training from scratch to learn to solve another task. The field of unsupereised zero-shot RL (Touati et al. 2023) aims to train a general agent, in a reward-fres pre-training stage, that is able to instantly adapl without any further training Lo any new reward function in transfer stage.
often define wide range of tasks via different
In theory, model-based RL alone achieve zero-shot RL when highly accurate model and unlimited deep decision-time planning budget are accessible. However, this is almost always impractical in real-world tasks. Model-free RL methods have been extended to perform zero-shot RL, where the general idea is to condition the policy on a task vector, w E Rd, that sunmarizes a task or policy, and train this policy to solve as many tasks as possible. This shifts the problem to that of finding a representative task vector encoding under a dimension budget Methods based on the Successor Features (SFs) framework (Barreto et al., 2017b 2018) linearly approximate the reward function from reward features (bs)) E IRd,, (s) R (s) w. The task vector then defines the contribution of each feature in the value of the reward, and the task-conditioned policy maximizes the features according to their weights. However, the zero-shot performance of such policies is known to depend on how well one can approximate the reward function of interest as lincar function of the reward features. This opens the problem of how to find good reward feature representation Φ, so that the set policies induced by is capable solving any task in a zero-shot manner. Current best- performing zero-shot RL methods often roly on the graph Laplacian, theoretically providing a Fouricr-like linear basis for the MDP (Barto & Mahadevan] 2003 Machado et al.) 2018). These methods have recently been bench-marked in extensive study by Touati et. al. (023)
2

---

Published in Transactions on Machine Learning Research (07/2025)
In this work, we introduce new method for identifying reward features for SF-based methods constructed by clustering the state space of the MDP. Based on this choice of reward features, we introduce Successor Clusters (SCs), a variant of SF's that predicts the expected time a policy spent visiting each state cluster. We use these to generate zero-shot policies. We show an overview of our method in Figure II SCs have important theoretical guarantees (as we list below), and provide state-of-the-arl zero-shot performance when compared to existing reward feature construction techniques used in SF-based methods. In summary, the contributions of this work are the following:
We propose a novel mathematical formulation to the problem of learning reward features in the SFs framework. We show that its solution bounded by the solution to the problem of clustering the state space of the MDP. This results in reward features defined by state cluster occupancy, Importantly, these features allow for high-quality reconstruction of any Lipschitz reward functions.
Based on this theoretical result, we introduce Successor Clusters, a novel type of SFss that predict future discounted cluster occupancics of policies. Additionally, we introduce a practical algorithm for unsupervised zero-shot RL based on SCs. We demonstrate that as the number and quality of clusters improve, the set of policies induced by features converges to set that contains optimal policy for any given task.
To idcntify state clusters, we introduce novel notion of state distance basod on temporal prozimity, under which states in same cluster are closer in terms of the minimal number of time steps necessary to reach each other. We introduce a practical algorithm to learn an associated latent metric space that encodes this distance function. We qualitatively show that this distance metrie, latent space, and clustering can be humanly interpreted.
Compared to existing methods, Successor Clusters (SCs) are highly interpretable and allow for intuitive visualization of the regions of the state space agent is likely to visit while solving given task. This stands in stark contrast to the often opaque, black-box nature of many competing approaches. The interpretability of SCs enables us to characterize their behavior in zcro-shot rein- forcement learning setlings in particular, SCs predict the espected lime spend in each cluster from a given state, conditioned on a reward function. We also demonstrate that this property makes it possible to infer reward functions basod on cluster visitation patterns or trajectorics induced by a policy. We believe this form of interpretability can positively impact research and debugging, while also improving transparency and trust in deployed systems.
We empirically show that SCs outperform state-of-the-ar SF-based methods in a challenging un- supervised RI. continuous control benchmark in terms of zero-shot performance. Additionally, we quantitatively show that our method leads better reward approximation performance compared lo stale-of-the-arl SF-based zero-shol transfer technique.
This work is organized as follows. In Section 2 we summarize the theoretical background necessary to introduce our methods. In Section g we introduce our theoretical results to the problem of learning reward features Φ, leading to the definition of SCs. We propose a practical learning algorithm for unsupervised zero- shot RL based on SCs in Section
4 We present quantitative and qualitative experimental evaluation results f we discuss how our work fits within the zero-shot research landscape and examine its limitations and potential improvements in Section 7
in Section 5. Finally, in Section
2 Background
Reinforcement Learning. In RL (Sutton & Barto 2018). the agent-environment interactions are gen- erally formalized via a Markov Decision Process (MDP), defined as a tuple M 2 (S,A,r.P, (42) whose elements respectively represent the state space, action space, reward function, state transition function, initial state distribution, and discount factor. The transition function p s) dictates the next state distribution, while the reward function r (s, a) dictates the next reward. We denote by S,, A,, and R, the random variables of the agent's 's state, action, and reward, respectively, at time step f. The goal of agent
3

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Reward Approximation 2 Φ W
Reward Features S:S+R S
Zero-shot policy
GF or USFA
T(a w.)
Set of policies:
Vw, xiaa 8; w) maximizes Øb
Figure 2: The central rolc of the reward features & in SF-based zero-shot RL. The reward features Φ : S + Rd are used to train a set of policies {i (a 8; wewcRR that maximize rewards defined as linear combinations Tw = Φ. w different choices of reward features & induce different sets of policies. During transfer, they are used to linearly approximate the reward function Φ wr. leading to the zero-shot policy (a 8; w,). Better reward approximation leads to better zero-shot policies.
is to learn a policy : S + A that maximizes the expected value of the return G, & Σx= 7 R The action-value function of a policy π is defined as q (" a) & Er S. = s, A, = a). where E, I- denotes expected value with respect the trajectories induced by ı and p.
Unsupervised Zero-shot Transfer in RL. In this work, we focus on a specific type of transfer in RL in which the dynamics are fixed but the agent has to adapt to any task defined by differont reward functions in a zero-shol manner. We assume a first stage of training where the agent interacts with a Markov Control Process (MCP), ie.,, an MDP without specific reward function, and learns some form of knowledge basis an unsupervised manner. The second stage consists of presenting reward function to the agent, either via the analytic function r as, or via set of samples {r(s;, at))= The zero-shot RL objective is to derive policy that maximizes the return of this reward function without any additional training or interactions with the environment. This problem is closcly related to Unsupervised Skill Discovery (USD) (Eysenbach) et al. 2019a Hansen et al. 2020), which aims to build a lask-agnostic skill set for use in any downsttream task. However, USD typically trains a meta-agent to use the skills during transfer.
Successor Features. Let Φ S x A + Rd be reward feature function (i.e., vector of d reward-like functions, also called "cumulants") Successor Features (SIss) (Barreto et al. 2017b) allow us to decompose the action-value function q", such that it is possible to instantly compute the q-valtes of policies given any reward function linearly expresile over & with weights w e Rdd
Tw (s o (s,a) = Nw (s, = 26" (s, (s,a) w,
where 26t (s,a) = E, 1 ( (Se+k, Ac+x) = s, a] are the SFs of policy n, and of ( a) is its action-value function w.r.t. task w. This result extends to the approximate scenario r R Φb w, where the performance w.r.t r of an agent maximizing Φ w improves with the quality of the approximation (Barreto eet al.] 2018). This decomposition is of critical importance because it allows us to perform generalized policy evaluation (GPE) (Barreto et al. 2020) efficiently, i.e., to evaluate policy & (s, = v6 as) w', Vw' E IR.. Similarly to an action-value function, aith follows Bellman equation and can be learned via any lemporal-differenc (TD) algorithm. Notice that in order to exploit the decompasition property provided by SFs, an agent needs to learn the SFs of a set of policies, so that it can use the most effective ones for any given we Rdd during transfer.
set of tasks;
Universal SF Approximators. Universal SF Approximators (USFAs) [Borsa et al. 2019 were pro- posed to deal with the fact that learning independent SFs, v6 a), for each agent's policy may be computationally prohibitive. USEAs draw inspiration from goal-conditioned neural network architectures, such as Universal Value Functions Approximators (UVFAs) (Schaul et al. 2015). and employ a single ap- proximator 26 (s, a; w) which is conditioned on task vectors w encoding the reward function. When using USFAs, the greedy policy (s) arg max 1p (8, a; w) approximates the optimal policy for rw, iee.
4

---

Published in Transactions on Machine Learning Research (07/)025)
Ow (s,a) (s, a; w) w. Crucially, USFAs aim to generalize over the space of weight vectors E R' in order to represent multiple optimal policies.
3 Successor Clusters as a Representation for Zero-shot Transfer
Our goal is to learn a good reward representation, S S + R4, that can be used with SF-based methods for unsupervised zero-shot transfer Intuitively, we aim to learn reward features that induce a good behavior basis for solving new tasks without additional training. Given reward features, Φ, such behavior basis is induced by learning policies optimized to solve tasks defined by linear rewards Tw = Φ w, for all w E W, where W C Rd denotes the space of linear reward weight vectors.
Critically, the performance of zero-shot SF-based agent is mathematically tied to how well can approx- imate rewards as linear function of the reward features. Let be arbitrary reward function, and let be the weight vector that best approximates r given features Φ: r = Iw Φ w. Then, assuming access to a policy Tw optimal w.r.t. Tw = Φ w, Barreto et. al. (2018) showed that:
|| g.- S ||| - woloooolo////////oooooooo (1- -7),
(2)
where of is the optimal action-value function for evaluated on r, and qft is the value function of Tw also evaluated on In other words, the optimality gap of Tw (i..., the difference between q' and qr= when evaluated on r) depends on how well the features & can be used to approximate work, our primary objective is to construct the best possible reward features, Φ(s) E R to be used with SF-based algorithms. We start by introducing a novel mathematical objective for the problem of identifying reward features in Section 3.1 Next, we connect this objective to a problem of clustering the state space of the MDP in Section 3.2). and finally introduce Successor Clusters (SCs), the main contribution of this work, in Section 3.4
Therefore, in this
3.1 A Formal Objective for Learning Reward Features
The problem of reward approximation is central to constructing features for successor features (SFs), since optimality is bounded by the quality of the approximation r R Φ w for any given reward function r, as shown in the previous section. Let r A frr r: S + [Tmin: maxl] denote the space of all reward functions bounded between Fmin and Fmax, and let &d e {Φ & S + Rd) denote the space of possible reward feature functions of dimension d. Intuitively, our goal (formally defined in Equation (3) is to identify a feature representation & E &d that enables broad and accurate linear approximation of reward functions in P. That is, we seek features that span the "widest" possible range of reward functions -enectiooseesuuinng that, across many possible rewards, & supports accurate linear approximations. Formally, this objective captures the goal of minimizing the expected mean squared error between each reward function in I and its best lincar approximation induced by Φ The error is computed over the state space S and the expectation is taken with respect to distribution P over reward functions likely to arise during transfer.
We call this objective reward coverage maximization and formalize it as follows
min ByeP(T) Φctd
min WERU
(bs)s)
(s).
(3)
Note that, in general, we do not have prior knowledge of the distribution over relevant or likely reward functions the agent may encounter. When the state space S is finite, reward function can be represented as a vector r E RISI, and P can be taken to be any distribution over R!SI, A natural and mathematically grounded srooice-sppccalllyy when little is known a priori about the structure of reward functions is to adopt a mazimum entropy distribution, which corresponds to the distribution that makes the fewest assumptions beyond basic constraints. For instance, if we only assume that the expected squared norm [|]]]]?]] is bounded,
We will restrict study stato-based rewards r (s') r (s, for simplicity. We note, however, that all definitions can be easily extended to employ general reward features Φ(s, a, s') instead. Withoot loss of generality, in continuous spaces, the sum over the state space can be replaced by an integral.
5

---

Published in Transactions on Machine Learning Research (07/)025)
then the maximum entropy distribution is multivariate Gaussian with zero mean and isotropic covariance: N((0,²7), Alternatively, if we assume that reward values are bounded (e-g., r(s) E [Fmin, ['ax] for all 8), then the maximum entropy distribution becomes the uniform distribution over the hypercube [rmins [max |8|
Both cases are useful in practice and provide principled ways to define expectations over reward functions while maintaining minimal prior assumptions
To the best of our knowledge, this is the first formal definition of an objective for learning features with the goal of approximating as many reward functions as possible in the SFs literature. The Fourier basis is an intuitive approach to this problem-it we can find an cquivalent for MDPs. Proto-value functions (PVTs) (Barto & Mahadevan, (2003) propose a solution to a very similar problem by finding a basis over value functions through an eigenvalue decomposition of the graph Laplacian. PVFs provide Fourier-like like basis functions that are smooth along the MDP geodesic directions. Figenfunctions are currently the main approach for learning features in SF-based methods. However, estimating them in high-dimensional spaces remains challenging. This has led to significant recent. research focused on improving Laplacian computation and decomposition techniques (Wu et al. 2019: Wang et al. 2021 Gomez et al. 2024). Critically, for any choice of feature basis, the dimension d is a significant constraint: current statc-of-the-an methods struggle to learn them when 100 (Touali et al. 2023 Therefore, in practice, we have a very limited budget on the number of features d.
The most important element for deciding basis Φ is the family of rewards I considered. In the extreme case, a valid reward E P has arbitrarily random values at every single state. Arguably, this type of reward does not correspond to any practical task, and thus, some constraints on I are acceptable. In other RL problem settings, works have proposed either piecewise or Lipschitz constraints in the reward functions to ensure that tasks make practical sense (Khetarpal et al.) 2022). To be best our knowledge, there is no similar consensus on subset of that is considered relevant and practical, but we think this discussion would be justified. In this paper, we will be focusing on K-Lipschitz reward functions: K--LiP = rr: : S + R Vs, E S, (r(s) r(s')) & Kō (s, (s, This assumption guarantees that rewards do not fluctuate too drastically w.rt.. some state distance metric 8.
3.2 Reward Features via Clustering of the State Spaae
In this section, we propose a novel approach to tackle the reward coverage maximization problem (Equa- tion (1) via a partitioning of the state space of the MDP. The proofs of the thcorems introduced in this paper can be found in Appendix A.II] We start by introducing a novel theoretical result that shows that, for K- K-Lipschitz reward functions, we can approximate a solution to the reward coverage maximization problem by solving a clustering problem over the state space of the MDP:
Theorem 1. Let C (C.) be portitioning of the state space S and let K- be the space of K-Lipschitz reward fanctions for some distance metric 5 over S. Let & (s) = [1c, (s),..,,l [s)] be the feature vector indicating subset occupancy; Ic, (s) if i E C. and 0 otherwise. Then, reward coverage marimization is bounded as follows:
Ep-P(Tx.2n)
min (r) 25 ($)) w) WERd SES
K
Σ (z. a.ECC.
(4)
Notice that the upper bound above depends on the choice of how to partition the state space, (C.) Minimizing this upper bound can be achieved by solving clustering problem whose goal is to identify clusters (partitions of S) that minimize the intra-cluster distances. Such families of problems can be efficiently addressed via standard clustering algorithms, e.g., K-medoids (Kaufman) 1990) or K-Means (Lloyd 1982) We note that we have a strictly improving bound as d increases: adding a cluster will always result. in a
II more general settings, where the state space is continuous infinite, P can be defined a distribution over functions. A common choice is use a Gaussian process (GP) prior, which generalizes the Gaussian case to infinitoddimensionall function spaces. If the reward function is assumed to lic in bounded interval, above, then a simple and flexible prior GP with constant mean function (e-g. (Fmin + Tmax)/2) and stationary kernel such the squared exponential (RBF) kernel with large length-scale. This setup encourages smoothness while avoiding strong structural biases.
6

---

Published in Transactions on Machine Learning Research (07/2025)
strictly better approximation, e.g., by splitting existing cluster in two. An important intuition on this choice of features is that, at any given time-step 1, Φ (St) indicates which cluster the agent is currently in: Φ is the cluster occupency vector. Additionally, when using this representation, the optimal w corresponds to a discrctization of the reward function over the clusters: wy is the average reward in cluster C₁ (sce proof in A.1 ). Since we assume Lipschitz reward functions, states that are identified in the same cluster by some clustering algorithm would likely have similar values of rewards. The restriction of the objective to the set of Lipschitz rewards is critical to this analysis. In Section [7.]] we further discuss this assumption and its impact the performance of the zero-shot policies induced by the reward features introduced in this section.
3.3 On Distance Functions for Clustering
In order to identify clusters Ci....,C,, Ca that induce features (() [Lc, (s). [s]] to minimize the upper-bound in Equation 4 need to define distance function 8(s, s') over states s' S. This raises the question: which state distance function 5(s, to use when identifying clusters C= [C;) via a clustering algorithm? In theory, the best distance function to use depends on the class of reward functions, F, being approximated. Distance functions that rely directly on the vector representation to be helpful in practice since they might incorporate implicit biases and are not linked to the topological properties of the stale space (e.g.; pixel representations, or (1c, cooordinnats in grid-worlds with obstacles).
a state are unlikely
We draw inspiration from stochastic shortest paths (Bertsekas & Tsitsiklis 1991). and propose to use a notion of distance based on the minimal number of time-steps it takes to get from one state to another. This is independent of the original stale representation and gives direct notion of connecteduess within the MDP graph. Spectral analysis of the graph Laplacian (Machado et al. 2018 Wu et al., 2019) and average commude time /Venkattaramanjjan et al.) 2019) compute such metries, but provide the expected time-steps under a fixee policy (e.g., a uniformly random or exploration policy). Instead, we aim to use the shortest commute time under an optimal goal-reaching policy. This better represents distances under efficient navigation policies, which we expect will be more useful for zero-shot transfer.
coal-conditioned RL allow us to compute such distance for any two states. Assuming any two states are connected (i.c., that for any two states, there exists a proper stationary policy), finding the shortest stochastic path between states and g is equivalent to finding the optimal policy from under constant reward- -1 and terminal state (Bertsekas & Tsitsiklis 1991). The negative value function then exactly corresponds to the distance function of interest: the minimal expected number time-steps to go from to g max, Uz (s: the optimal value function conditioned on goal g. Based on this idca, we introduce the average ménimmal poth distance, symmetric state distance function defined
1 d. (81, $2) A max U (81; - max Un 81).
1
(5)
2
2
where Ux ($1; $2) is the value function of π for state 81 with goal (terminal state) $2. Using the distance d, defined above to minimize the upper-bound in Equation 4 has implications on the reward space I and the resulting clustering. A K- K-Lipschitz reward function under d, implies that the reward difference between any two connected states will always be lower than K. Identifying clusters that minimize Equation (4) under d, establishes a notion of temporal proximity: states within the same cluster are reachable in a short number of time steps. In Section 4.2 we will discuss how to approximate d, in practice.
3.4 Successor Clusters: Predicting Future Cluster Occupancy
Based on the theoretical motivation for using reward features Φ(s) defined via partitioning the state space of the MDP, introduced in Theorem [ we are now ready to introduce Successor Clusters, the central contribution of this work:

---

Published in Transactions on Machine Learning Research (07/2025)
Given a partitioning C = {Ci)i= of the state space S, the Successor Clusters (SCs) of a policy и for a state-action pair (s, a) defined as:
(s, = E
Σ x 0==
s A where ((s) = [1c, (s). ,c,(ss) (6)
It follows from choice of & that each i-th component of the SCs vector 30 (s, a) R i.e., (s, I (M,
(s.a)..... a).... (s, represents the expected discounted time spent visiting each cluster C; when following policy π.
Notice that SCs are a type of SF whose values are naturally more in- terpretable. Intuitively, since 2h characterizes the discounted mum- ber of steps spent in each cluster C,, it also encodes the expected amount of time spends in each region of the state space. In Fig- ure 3 we depict an example of the SCs of a policy to illustrate this idea. SCs therefore enable us to easily interpret the outcome of a policy without needing to run it: while does not detail the or- der a policy visits each cluster, it summarizes the policy in terms of spatial occupancy. This interpretability can be an advantage when employing SCs. When learning an optimal policy, it can help derstand its trajectory and inform and task. When learning function approximator, it can be easier to detect, identify and correct errors, since we have more intuitive understanding of what outputs to expect.
C
C2
11
0
C3
2
0
S0
4
5
the geometry of the domain
The Successor Clusters as- sociated with the policy in black are [0, 0, 4, 2, 11]. and consider a finito-horizon problem in which cach episode lerminates after fixed, pre- determined number of steps.
Figure 3:
sb" (so, (8o)) We assume y
We also note that E vhg (s, a) = I-7 for any given state-action pair (s,a). Hence, we can see SCs as type of occupancy mea- sure (Laroche Tachet Des Combes 2023) defined over regions of the state space instead of single states. Notice that the Successor Representation (Dayan 1993] SR) of a policy म, v ss,,),, is defined the expected discounted number of visits Therefore, it is also possible to interpret SCs generalization of the SR. We formalize this notion in the following observation:
particular state s' if an agent starts in and follows
Observation 1. Let M be an MDP with discrete state space S. For given policy let the Successor Rep- resentation be x6 (s, 5') = Ex Eh tht ((5)4)11=0)) S =]. By slight abuse of notation, the SC for a cluster C is 3pth (s, Ca)= Ex [ER= I St =]]. Therefore, we have that 1/2 (s, =.)== th" (s. s'). It follows that as the number of clusters increases (d + (S]), the SCs and the SR of a policy become equivalent, i.e., as [Ca) + and C. + {si), (s, + (s, 8t).
Obscrvation immlics that we can interpret SCs as a form of spatial generalization of the SR, modulated by the number of clusters, d. This result also exlends to continuous states by replacing the surmmation with integrals, e.g., (T, Ssee s (s,ss')ds
Recall that main goal is to perform zero-shot transfer. To accomplish this, SF-based methods learn set of policies <[als; w), for all w, capable of maximizing any linear combination of the reward features (Section D Then, given any reward function the agent can follow, in zero-shot manner, the policy 7 (als; wr), where s & w,. When employing the SR instead of SFs, Alegre et al. (2022) showed that this process induces an optimal policy for any reward function. Since according to Observation [ SCs are generalization of the SR, they also benefit from similar properties. In Appendix A.2 we discuss this theoretical connection and show that SCs also produce an optimal set of policies as the number of clusters increases.
4This follows directly from the fact. that with one-hot vector features, Ed (s) and 5 7" ====1
8

---

Published in Transactions on Machine Learning Research (07/2025)
Another relevant property of the cluster- based reward features used by SCs is that they are orthonormal and, therefore, of full rank d. (Touati et al. (2023) analysed the feature rank of different methods for learning reward features. They showed that methods that produce features with a rank lower than d (which they refer to as feature collapse) have lower zero-shot performance. We note that independent and orthogonal features such as ours are likely easier to approximate and predict by function approximators, especially the context of SFs.
4 A Practical Successor Clusters Algorithm for Zero-Shot Transfer
In this section, introduce a practical algorithm, based on the definition of SCs and their theoretical motiva- tions presented in the previous section, for unsupervised zero-shot transfer in RT. In particular, we introduce an offline deep RL algorithm that learns SCs in an unsupervised manner (i.e., without information on the distribution of testing lasks) using only a dataset of offline experiences D = {[8,, ai, s()]==- In summary, given a dataset of experiences D = {(s, a,, s;)) our method learns (i) a USFA modeling the Successor Clusters, (s,, a; w), and a corresponding policy network <(s; w) (Section : and (ii) a latent space projec- tion function, (s)); in conjunction with goal-conditioned action-value function, Q(8, a; g), with the latter guiding the construction of the former (Section 1.2 Reward features ((s) [1c, (fp(s), 1c, (P(s)) are constructed using online K-Means (Section
4.2 All above-mentioned components are learned as the agent Φ with weights w sampled (without loss of generality) from The algorithm's pseudocode is presented in Algorithm
optimizes reward functions of the form
W (w& E [dd ||w]22 = vdj (Section ion 4.3
and is
discussed in detail in the following sections.
Algorithm 1: Offline Successor Clusters
Input: Datasct of offline transitions, DD {(8;, Gs fm
Initialize the following modelss Successor Clusters (b88, a; w) (Section 4.1 policy (s; w); reward features Φ(s); projection function (ss) onto the latent space S (Section 4.2 ; and goal-conditioned
action-value function Q(s, a, g) (Section 4.2
while Thaining budget lasts do Sample minibatch of transitions {(ss, a, (,, +i12 ~ D and weight vectors (w;) W (Section 4.3
Update goal-condition tioned Q and latent features s. Recompute clusters &
4 Update goal-conditioned action-value function Q (s, 0; via a goal-conditioned RL method
Update latent space projection function with Lp (Equation (8)
Update cluster featurcs using datapoints y (s) via online K-Means
Update SCs vp and policy
Update vb (Equation (7)) and (Section 4.1
4.1 Training the USFA
Since we would like to learn SCs that generalize over a range of different reward functions (each encoded by different weight vector w), we employ a USFA ub(s, a; w) = ght (s. a), approximated via neural network. Recall that Successor Clusters (b(s, a: w) represent the expected discounted time spent in each cluster when solving a given task w by following policy rts; w). In discrete-action domains, the policy (s;; w) can be directly inferred from the SCs: T(:; w) E arg abs,,; A USFA can be trained with DQN-like updates:
C =E S.a.'' [I)(s) (s) + "sb.b''s (s',a":w) 2 (s. (s,2:))) where a" = =sss:w E arg max (s',a':w) wwWCR"
a'EA
(7)
In contimuous-action domains, the actor policy (s; w) E arg max 1b a: w) w can be optimized via deterministic policy gradients, as in the TD3 algorithm sampling tasks w in W C Rdd will be detailed in Section 4.3
Fujimoto et al. 2018). The proposed approach for

---

Published in Transactions on Machine Learning Research (07/2025)
Note that the agent can be in only one cluster at time: Φ is a one-hot vector, so Σ Lk=1 Qb (s) 1. This means that for fixed discount 7, the total expected discounted "visit time" given by the Successor We can exploit this property to constrain the USFA neural network to only produce outputs that satisfy this property. For instance, an agent should not be able to predict it will spend 150% of its time cluster C1. or -00 in cluster C2: outputs should always be positive and sum up to 100%. This can be achieved by adding a rescaled softmax layer after the final layer of s, constraining it to produce outputs whose sum is 1/ 7))
Cluster representation is constant: for any s, a and w, Σm= the (s, (s,a;w a) = E 70] = (1-11117))
4.2 Learning the Minimal Path Distance Function
We now introduce a method for learning the state distance function, d., introduced in Equation ( to use when partitioning the state space. Note that this is proper distance function and can be used for clustering the state space S. However, it cannot be used with traditional clustering methods such as K-Means, as their convergence guarantees hold only for Euclidean spaces. To overcome this problem, we construct a latent space projection şp : S + S C R where, by construction, the latent Euclidean distance ā = =I-112 between (s1)) and ((82) matches with the average minimal path between 81 and $2 in the MDP, under the original distance function du Fouss
al. (2005) showed that, for undirected graphs, there exists a latent space where Euclidean distances in the latent space correspond to the square root of the average commute time in the graph. This allows us to define the following loss function to learn y::
(8) Computing this loss only requires to sample pairs of states from the MDP. However, the crux of the effort is in learning d, through goal-conditioned RL as per Equation (5). We rely on existing goal-oriented RL algorithms, especially the Hindsight Experience Replay (HER) (Andrychowicz et al. 2017) technique, to
Li - Esg (0. - d. (s.9))²
estimate max, ((, azg) x max, Ux (8; This leads to the final loss for 4,
Li = E. (110(0) (11)())-v())l>> (1)l22 2 max Q (s, max Q (1))
1
27
Such goal-reaching agents can be trained entirely offline with architecture and algorithms virtually identical to USFA, which limits hyperparameter search. We then compute clustering {Cr)= of the states in the latent space, vpS),, using K-Means. The resulting rewards features used in our method are then ((s) = [1c ... (.cc(((())) leading to the SCs method described in Section 3.4
4.3 Reward Function Approximation and Task Vector Normalization
We want to compute the best linear approximation with weights w given a reward function Section 3.2 established that the optimal w for cluster occupancy features corresponds to average cluster rewards, w [r (Ci), ,r ((Ct),".,,rCCa)))) (Ca))", with each r rCi) defined as the average reward in the cluster C. More concretely, consider samples {sy, r(s))} from an arbitrary reward function r. Let X, be the set of states from the samples that belong to cluster C. Then, the weight vector w that best approximates this reward can be efficiently computed the average reward per cluster:
wi RM
Σ (8)), (1,,) 2EX,
(10)
The fact that the task vector is expressed entirely in terms of the reward function is of particular relevance. Indeed, any reward function has two key properties that can be exploited: first, it is independent of scaling: the optimal policies do not change if the reward is scaled by a positive factor a. Second, a continual environment (with no terminal states), the reward function is independent of translation: the optimal policies do not change if the reward is translated by constant 3. We can use this to our advantage prevent USFAs from learning "redundant" task vectors, i.e., vectors that share same optimal policy. For
10

---

Published in Transactions on Machine Learning Research (07/2025)
example, tasks w w= 1-1,1,,, w [1, and [0, 0.33, 0.66) all share the same optimal policy. We opt to normalize all task vectors onto the sphere, vd(w (w w)/ ||wll, before passing it to the USFA network. That is, without loss of generality, we define W {w Rd ||w||2 vd). We also randomly sample task vectors from this sphere during training simulate different reward functions.
4.4 Following Provided Trajectory
The interpretable nature of SCs has one more key advantage: we can it for imitation learning, i.e., to train an agent to imitate given trajectory. In RL, reward design is major problem since in many cases, designers can characterize the overall behavior the agent should produce, but creating a corresponding reward that induces this behavior is not trivial. The agent will often take shortcuts, find local optima, find loopholes to ignore the desired objective entirely. With SCs, we can reason in terms of expected cluster visits or analyze a trajectory terms of expected visits, which can, in turn, be mapped to a task vector w. First, assume that a human expert wishes to generate a trajectory T 8p, $1, sT. This can be mapped total cluster visits occurring during the trajectory: O (so) = Σ [4c, (st).... Lc, (s:)] (ignoring the discount y here for simplicity). We can alternatively assume the expert knows which clusters they would like the agent lo visit, and that the associated visitation count is the diameter of the cluster. These desired visits can then be used to find the vector w with which the expected visits predicted by the SCs will be closest to the desired visits:
w - arg min b (So, a.; w) (su)ll?
(11)
where a+ arg max 1b (8, w) This objective can be easily optimized with gradient descent. During this process, w can be constrained to be on the Vd-radius sphere (see Section 4.3
5 Experiments
In this section, our goal is to empirically evaluate the Successor Clusters method and to empirically validate its theoretical properties. In particular, we aim to igale the following research questions: Q1: Do SCs result in better zero-shot performance when compared to other methods for learning reward features (Sec- tion 5.1 ? Q2: Do SCs result lower reward prediction error, as suggested by Thcorem [ (Soction 5.2 Q3:
Can we generate interpretable visualizations of the different learned components of our method (Section 5.3
5.1 Evaluation of Zero-Shot Transfer Performance
Figure 4: Domains used for evaluating zero-shot transfer performance: Quadruped, Walker, and Cheetah robots from URLB (Laskin al. 2021). Transfer tasks involve performing movements such walking, running, jumping, and Nipping.
We follow the experimental setting employed by Touati et al. 2023 which comprises robotic tasks from the Unsupervised RL Benchmark (URLB) [Laskin et al. 2021 In this fully offline sctting, agents have access to an offline replay bufler containing 5 x 10t experiences collected using an exploration policy trained with Random Distillation Network (Burda et al. 2019 as provided by the ExORL repository (Yarats et al., 2022). Agents are then tasked with learning reward features and SFs in a purely unsupervised manner with a budget of 109 gradient updates. The method's hyperparameters and other experimental details are specified in Appendix C We evaluate the zero-shot performance of each method in the Quadruped, Walker, and
11

---

Published in Transactions on Machine Learning Research (07 (025))
Cheetah simulated robots from URLB, depicted in Figure 4 The performance is evaluated w.r.t. different reward functions that the agent approximates using their learned features (Section 4.3) e-g., rewards for walking, standing, and jumping. All reward functions are described in Appendix C.2
We compare Successor Clusters with the Laplacian representation [Wu et al.) 2019 since this is the approach that generated the best overall zero-shot performance when used with SFs in the extensive evaluations conducted by Touati et al. (2023). Laplacian eigenfunctions provide a Fourier-like basis that span value functions of an MDP, as detailed in Section 33.1 We also compare employing SCs using Laplacian features the latent space &p used to define the distance function used for clustering instead of the average minimal path (Minpath for short) introduced in Section 3.3 as well as Minpath latent space features for completeness. We used = 50 as the dimension of the reward features ((s) E Rd, similarly to what done in Touati et al. (2023). Recall that for SCs, corresponds to the number of clusters. In which follows, we normalize returns and show the percentage of the performance w.r.t. the best-encountered return over all agents on each task. We depict the mean and 95% stratified bootstrap confidence intervals over ten runs with different random seeds (Agarwal et al. 2021
SC + Minpath
SC + Laplacian
Minpath
Laplacian
Qu Quadruped
Walker
Cheetah
800
Return
(000
Return
600
Return
600
00.0
400
200
Walk
Ron
Jump
Stand
Walk
Fon.
Flip
Stond
Wallz
Walk Dackward
Ron
Rum Backward
Task
Task
Task
Figure 5: Mean returns obtained in the Quadruped, Walker, and Cheetah robotic tasks of the URLB when using Successor Clusters (SC); or method, depicted in shades of red, or Successor Features (SF); depicted in shades of blue, with Miupath or Laplacian reward features.
Overall Performance
Walker Performance
SC + Minpath
11 25-
SC Laplacian
11 50
Minpath
= 100-
Laplacian
n = 200
0.45
0.60
0.75
0.45
0.60
0.75
0.90
Normalized Return
Normalized Return
Figure 6: Overall performance (mean normalized returns with 95% stratified bootstrap confidence intervals) in the Quadruped, Walker, and Chee- tah robotic tasks depicted in Figure
Figure 7: Ablation of the dimension E (25, 50, 100, 200) of the Minpath latent space p(s) E R" used by SCs in the four tasks of the Walker domain.
In Figure 5 we show the mean return of each agent obtained in each task and robot. We note that agents using SCs (red bars) very often outperformed their "raw" features counterparts (blue bars). Agents using solely Minpath reward features, in general, did not perform well in this benchmark. This is to be expected since they were not specifically designed to linearly approximate reward functions. Additionally, in Figure G the overall performance of cach method averaged over all tasks. Notice that our method, SC + Minpath, not only obtained the best overall performance, but that combining SCs with other lypes of reward features (e.g., SC + Laplacian) results in improvement when compared to using the original reward features directly (Laplacian). These results imply positive answer rescarch question Q1 and show that SC's theoretical properties (as motivated by Theorem I ensure that agents perform better when tackling tasks with previously unseen rewards in zero-shot manner.
show,
12

---

Published in Transactions on Machine Learning Research (07/2025)
In order to investigate the impact of the dimension n of latent vectors (o(s) R", we conduct an ablation experiment in the Walker task whose results are depicted in Figure 7 Notice that using too low values of n (e.g., 25) hurts performance, possibly due to the dimensionality not being sufficient to encode distances in the high-dimensional state-space of the Walker domain. Similarly, too high values (e.g., 200) led to decreased performance. The best performance was found for intermediate values of n, bet ween 50 and 100.
Additionally, in Appendix C.3 we include comparison with the forward-Backwarr (FB) representation method (Tonati et al., 2023), state-of-the-art approach for unsupervised zero-shot transfer in RL. While our primary objective is to improve upon Successor Feature- based methods by designing more effective reward representations, our approach also performs competitively when compared to FB. Importantly, please note that FB adopts a fundamentally different. strategy for zero-shot learning and does not belong, strictly speaking, to the class of techniques we aim to improve upon. Specifically, unlike our method, (i) it does not leverage a. linear decomposition of reward functions; and (ii) it focuses on learning features that support the roconstruction of optimal g-functions, rather than features that enable accurate lincar approximations of a broad range of reward functions.
5.2 Evaluation of Reward Approximation Quality
To investigate Q2, and Q3, we perform qualitative and quantitative experiments on the Maze environment, as it allows us to more easily visualize the agent's ' behaviors. We did not include this domain in the validation of Q1 since all algorithms had an identical (ncar-optimal) performance on the goal-reaching tasks defined in this benchmark. The Maze state space consists of the location of the agent in a 2D square space (z. y) as well as its velocity along each dimension (4r ú); for a total of 4 inputs. There is a cross-shaped wall in the middle of the environment (shown in black on all figures), leading to non-trivial geodesic distances. Actions are 2D vectors defining the acceleration of the agent along the (z, y)-plane. We follow the assumption from Touati et al. (2023) that possible reward functions depend on the agent's position and are invariant with respect to its speed, that is, the family of possible rewards is r = fr r (I, y, à. i) = r v)]. Clusters are defined over the 2D position plane. The environment has a long horizon: it takes around 400 steps for an agent to move from one corner of the square to the other.
We start by comparing the reward prediction error of SCs and Laplacian w.r.t. the reward coverage max- imization objective (Equation 33) considering different families of reward functions. We also study the impact of the hyperparameter d, the number of reward features, on each of the methods. We evaluate the agent using four different families of reward functions, each of which represents relevant tasks that an agent could be faced with during transfer. The reward familics are (i) goal-reaching rewards, each a single highly-truncaled Gaussian placed al the 20 goal locations from the benchmark; (ii) trajectory rewards, which follow a path along the MDP with increasing rewards leading up to some goal state; (iit) Gaussians, a sum of to 20 Gaussians with random means, standard deviations, and weights and (iv) smooth functions combinations of linear, trigonometric and exponential functions. We show a visualization of all these reward C.5L For all of these reward families, we compute the best linear approximation w as minw Es (r (s) (s) w) and show the resulting mean squared error (MSE) in Figure g
functions in Appendix
In Figure notice that SCs lead to a significantly lower prediction error when compared to Laplacian features when evaluated in all families of reward functions. Although the goal-reaching reward functions are not Lipschitz, their discontinuities are local, so the performance of SCs is not significantly impacted. This is an important observation: although Theorem [ holds for Lipschitz rewards, the piecewise nature of the SCs reward approximation makes it strong option for sparse rewards. In comparison, Laplacian features consist only of smooth functions, so the corresponding approximation struggles with such sparse rewards. We further discuss the Lipschitz assumption and the consequences of violating it in Section 7.1 As discussed in Section 3.2] our dluster-based reward features are guaranteed to approximate reward functions increasingly better as the number of features, d, increases. In this Figure g we empirically demonstrate this property, and observe that it does not hold for the Laplacian-basod features. This may be due to the fact that latter eigenfunctions correspond to "high-frequency" functions.
13

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Laplacian (d = 25)
SCs (d 25)
0.6
Laplacian (d 50)
SCs 50)
Laplacian (d 100)
SCs (d 100)
Mean Squared Error (MSE)
0.5
0.3
0.2
0.1
0.0
Goul-reaching
Trajectory
Gaussian
Smooth
Reward Class
Figure 8: Mcan squared error (MSE) of the reward approximation of SCs and Laplacian for various familics of reward functions and a varying number reward features, d.
5.3 Evaluation of Successor Clusters Interpretable Representation of Behaviors
Finally, we investigate Q3 based on the interpretability associated with our method by providing visualiza- tions of the learned latent space yo, learned clusters {Ci) and the SCs predictions given by the USFA network (bss, a; w) in the Maze domain.
Figure 9: Left: Learned goal-reaching Q-values, Q(:g), for a given goal state 9 (red circle). Middle: Associated latent distance, ||s (s) - & (g)ll2 Right: Partitioning of the space into d = 50 clusters with associated diameters depicted in red.
Minimal Path Latent space. discussed in Section 4.2 our method (Algorithm trains a goal- reacher policy and use its Q-value construct latent space representation where the distance between any two states matches the minimal number of time steps necessary transition from one to the other (Minpath distance). In Figure g we visualize the Q-values, Q (: associated with reaching goal state and the corresponding distance in the latent space, ||ye (s) ip (g)ll2- Notice that the distance function is lcarned effectively since it correctly accounts for topological propertics of the environment, such as the geodesic distances induced by the presence of obstacles. Figure IO shows (-SNE visualizations of the latent space, similarly as done by TTuati et al. (2023). We can observe that the latent space learned by Minpath is consistent with the structure of the environment's state space, with states separated by obstacles being projected onto corresponding distance embeddings. In comparison, the learned Laplacian representation does not result in easily interpretable visualization of the MDP dynamics.
14

---

Published in Transactions on Machine Learning Research (07 (07/202) (025)
State
Lap
Minpath
Figure 10: t-SNE visualization of the learned latent space (Raw state, Laplacian, Minpath).
Clustering. In Figure (rightmost panel), we show a visualization of the learned clustering over the Maze state space. Although the state distribution in the oflline replay buffer is strongly biased towards the upper-right region of the maze (where initial states are located), the partitioning of the state space is not significantly biased, and the various regions of states are fairly evenly covered. We also show the radii of each cluster-most most clusters have diameter of around 70 time steps, showing that Maze is quite long-horiag environment. The topology of the environment also captured by the structure of clusters, as the clusters are located around obstacles and do not overlap with them.
(a) Left: Reward function being approximated (ground truth). Right: Associated reward approximation. The trajectory executed by following the agent's policy is shown red-yellow.
(b) Predicted cluster occupancy induced by the reward approximation after three steps along the trajectory following the agent's policy (red-yellow curve).
Figure 11: Visualization of the reward function approximation and the cluster occupancy predicted by SCs.
Successor Cluster prediction. We now show a visualization of one of the central elements of this pa- per: the cluster visitations predicted by the SCs. In Figure [1] we visualize the complete zero-shot transfer process implemented by our method. First, in Figure 1ia an arbitrary reward function discretized into clusters by identifying the weight vector that best approximates it, i.e., Φb w, following the
15

---

Published in Transactions on Machine Learning Research (07 (075))
method introduced in Section 4.3 Next, the SCs USFA predicts the future cluster occupancy of the op- timal policy w.r.t. this reward, and follows the corresponding trajectory (Figure libj. Since Maze has a long horizon, the agent only predicts the next few clusters it will visit due to small values of of as in- creases. We provide an animated version of this prediction, as the agent advances through the maze, at /vinn/viwwwaweeee wtt.r/cowwovviivuuussssoooooo--e wwwp:/essooorcuussteiiisssseeeee
This prediction can be used to enable or im- prove several human-AT feedback loops. First, from a rescarch and development perspective, users can have better understanding of the network's error and more easily detect implementation mistakes. Second, for safe RL applications, can study the agent's expected visit of undesirable clusters and adapt the reward function and training accordingly. Third, this also allows the users to better understand the intent of the agent and the impact of different tasks w on the generated trajectory, without needing to deploy the policy, Finally, the intuitive nature of cluster predictions makes it a better interface for users than commonly used black-box features.
Figure 12: Controlling a SCs agent by changing cluster reward weights, w, to produce trajectory along the top or boltom regions of the maze (left and right panels, respectively) and that reaches a target cluster in the bottom-right region the environment.
Controlling the agent. Our method's USFA is trained to navigale to various clusters, that is, the agent is able to traverse or avoid clusters C, based on their corresponding individual values, w;- By adjusting w, we can thus control the expected trajectory executed by the agent over clusters states. We visualize this capability in Figure Here, different choices of control whether the agent should navigate along the top bottom of the maze when reaching larget cluster in the bottom-right region of the environment.
Using SCs induce agents to follow a desired trajectory. Left: Specification of the time a designer would like the agent to spend in each cluster. Middle: The vector optimized to generate these visits from the USFA. Right: The effective USFA predictions along with the agent's generated trajectory.
Figure 13:
Fitting a desired trajectory. Finally, as discussed in Section 4.4 our method allows for a given a trajectory of interest to be expressed in terms of desired cluster visits: O (se) = Σ [A,, (st), 1c, (s:)] Then, it able to identify a task vector w that expressos a reward function that induces the desired trajectory. While this enables agents perform behavior cloning, one can also provide a trajectory that the designer does not know how to achieve. In Figure 13 we employ this idea to induce agent to reach the
16

---

Published in Transactions on Machine Learning Research (07/)025)
closest corner of an obstacle depicted in yellow (leftmost panel). This results in trajectory where the agent spends a few steps in each of the clusters along the way towards the obstacle, and the remaining steps of the trajectory in the corner cluster. This gives us the desired visits from the starting state, O (so). Next, our method computes w through Equation (11b via gradient descent. This process identifics the vector w that is locally most likely to result in the desired cluster visits, +b (s0, a.; w) R O (so)- Finally, we can then execute the zero-shot policy for w to obtain the desired trajectory. Note again that this procedure is qualitatively different from behavior cloning. In particular, we only need to provide approximate cluster occupancies, but we do not need to inform the agent about what actions it should take.
6 Related Work
In this section, we discuss existing relevant. approaches for learning reward features, as well as other related works in the literature on successor features, discrete representations, and unsupervised skill discovery.
SFs and reward feature learning. Previous works have lackled the problem of learning a set of re- ward features to be used within the SFs framework. Given a set of training tasks, [aarreto et al.) 72017a) 2018); Carvalho et al.] (2023b (a) construct reward features via linear regression with the goal of linearly approximating the reward functions in this training set. This type of approach works well when a set of representative training tasks is available for training, with the assumption that test tasks share structural similarities with training tasks. We, by contrast, tackle the unsupervised setting in which an agent has to learn reward features without access lo a predetermined distribution of lasks. Hansen el al. (2020) proposed to learn features via a behavior nuutual-informatin vector plays both the role of the "skill" vector (typically employed in mutual information RL algorithms) and the role of the linear reward weights. Touati & Ollivier (2021); Touati et al. (2023) introduced the forward-backward representation, which models low-rank approximation of the successor measure of a pol- icy. Although these approaches work well in practice, they do not produce representations that are naturally interpretable, such as the ones that underlic our method. Reinke & Alameda-Pineda (2023) studied the problem of extending SF-based methods to solve tasks defined as non-linear functions of reward features. However, they assumed that the reward features were provided a priori and did not tackle the problem of learning expressive reward features. Another approach designed to optimize reward coverage maximization, although not under the SF's framework, was introduced by Frans et al.) (2024) and consists in approximating randomly sampled reward functions. However, this approach is very computationalyy intensive. Moreover, it requires manually-designed specification of the distribution of training reward functions, for which no theoretical guarantees are given with respect to their ability to generalize to arbitrary test reward distri- butions. We, by contrast, characterize and construct a particular choice of Φ and w with strong relevant thcoretical propertics. Other works have explored the problem of learning a set of optimal policies given predefined reward features (Zahavy et al., 2021 Alver & Precup 2022a Alegre el al., 2022) an important problem orthogonal to the one investigated in this paper.
objective (Eysenbach et al.) 2019b) in which the weight
Unsupervised skill discovery (USD). USD methods aim to learn set of policies that can be used to solve unseen downstream tasks. USD hot generally try to create policies that reach key states (Machado
et al.
2018 Jinnai et al. 2019 or that maximize skill diversity through a mutual information objective /re-
gor et al. 2017 Eysenbach et al. 2019b). In the context of Successor Features, Hansen et al. (2020) intro-
duced state features associated with the probability of each skill visiting a given state. Eysenbach et al. (2022)
have shown, in a geometric study of USD, that diversity- basod methods ignore regions and associated skills that are too close to the sampling distribution, since these may be difficult to differentiate. In comparison, SCs allows agents to evenly reach the entire state space being approximated by state partitioning.
Discrete representations. Grimm et al. 2019) proposed using a set of disentangled features that can be controlled independently by discretizing domain features into a set of discrete bins. However, these can only be employed in domains in which it is possible to specifyy tasks in terms of disentangled transformations. Morcover, they assume that the order of actions exccuted by the agent does not matter, which is typically not true in RL. Ramesh et al. (2019) also employ a clustering approach in the context of the successor representation framework. However, their approach is significantly different. than ours. They, for instance,
17

---

Published in Transactions on Machine Learning Research (07/2025)
cluster the state space using the SR distance metric and use the clusters sub-goals to define options. We, by contrast, cluster the state space w.r.t. the minimal path wet states and use clusters to define reward features used to construct SFs. Our work also shares interesting connections with reward-predictive representations [Lchnert et al., 2020 2022). In particular, Lchnert et al. (2022) defined reward-predictive clustering as a state abstraction that groups states predicted to produce the same future reward sequences. Although they show that their learned representation (given a training task) can be used as an initial solution to accelerate the agent's 's adaptation to novel reward functions, the proposed framework is not capable of performing zero-shot transfer. We, by contrast, construct a reward clustering representation tailored for zero- shot adaptation. Farebrother et al. (2023) define successor measures with indicator functions in a way that resembles SCs (Equation 6 However, they use random indicator functions (or random clusters) with the goal of deriving auxiliary tasks in a single-task setting. We, instead, learn clusters with properties relevant to the goal of approximating reward functions in the multi-task zero-shot setting. Similarly to our reasoning, they argue that binary features can be beneficial since they are casier to adjust compared to real-valued functions. Another form of discretization used in RL with function approximation is tile coding (Sherstov & Stone, 2005) (Sutton& & Bartol 2018). In tile coding, the state space is split into regions visited under indicator function. Our method has key differences when compared with tile coding; (i) the amount of features in & is strongly limited, making it difficult to layer tile maps; (it) the bounds of the observation space are generally not known in advance (or are very large), making it difficult to define tiles; (6ii) with limited features, off-th tile coding may tile regions of I that are not part of the state space S, This is a limitation avoided by directly clustering states in S, as in our work.
7
Conclusion
We now discuss a few limitations of our approach, and then summarize our theoretical contributions, em- pirical findings and comment relevant future research directions.
7.1 Limitations
Curse of dimensionality. In dine-sdimesiioall spaces, the required number of clusters, d, can be high due to the curse of dimensionality. From a practical standpoint, it is important for all prior knowledge about the reward space of interest, r, to be leveraged to reduce the space being partitioned. From a theoretical perspective, we expect it should be possible to borrow ideas from the tile coding literature to layer cluster maps. In particular, with d binary feature maps, one should be able to define up to 2d regions of the state space. Practical algorithms for achieving this (and learning the associated USFAs) could substantially improve the scalability of Successor Clusters.
Lipschitz reward functions. A main limitation of our method is generalizing over non-smooth reward functions. We restrict our theoretical analysis (Theorem I to the space of Lipschitz reward functions. We now briefly discuss how to relax this constraint to local Tápschitz functions. Let r be locally Lipschitz on $ where Š is the sot of states in the region where the Lipschitz property is not satisfied. For clusters Cs remains. For clusters Cg that contain states of Š, the
that have no state in Š, the guarantee of Theorem [ guarantee is lost. In sparse reward environments, $ is quite small, which is why the method still performed well on the goal-reaching tasks (Figures S and discussion in Section 5.3). With Successor Clusters, this loss of guarantee on Š is interpretable under the observation that the policy cannot fine-tune within clusters. It might then either underestimate cluster, avoiding a high-value part of the cluster because of the low-valued rest, or overestimate it, crossing a low-valued part of the cluster because of its higher average value. To the best of our understanding, non-sipschitz reward functions that are not sparse are quite rare, so SCs should generally be able to produce strong zero-shot policy.
Exploration and sampling state distribution. All the experiments in this paper were performed in an offline RL setting using the RND exploration policy dataset provided by URLB (Laskin et al. 2021). Offline RL is fundamentally limited by the quality of the datasct: not much can be learned about regions of the state space that are not covered in the datasct. For example, in the Maze domain, some exploration policies never make it past the first room. Hence, we should not expect zero-shot methods to be able to
18

---

Published in Transactions on Machine Learning Research (07/2025)
reach there. However, Successor Clusters dependent on the sampling distribution in clearer ways. In Theorem ] our sum is defined over the state space, but in practice, we only have access to an expectation over the (unknown) sampling distribution. Concretely, this means that parts of the state space that have a higher donsity than others will produce more clusters, leading to imbalances in the state space partitioning and zero-shot policy performance across the state space.
7.2 Summary and Future Work
In this work, we introduced Successor Clusters (SCs), a novel mothod to tackle unsupervised zero-shot RL problems. Our approach leverages a novel mathematical formulation to the problem of learning reward features capable of spanning families of reward functions, and that are (by design) tailored for use within the SFs framework. We formally characterized how reward features can be constructed by partitioning the state space and mathematically quantified upper bounds their capability to approximate any Lipschitz reward function. To construct partitions of clusters, we introduced a novel notion of distances in an MDP based on the minimal number of time steps to transition between pairs of states. We showed that such distance metrics can be learned via goal-conditioned reinforcement learning mods. By employing cluster occupancy features, SCs are capable of predicting the expected discounted time that a policy spends in each state cluster. After a pro-training phase, our method is capable of approximating and maximizing any new reward function in a zero-shot manner, i.e., without requiring any additional learning. Importantly, we formally showed that as the number and quality of clusters improve, the set of policies that can be induced by SCs converges to a set containing the optimal policy for any given task. We empirically demonstrated that our method outperforms state-of-tho-art competitor in challenging continuous control benchmarks, both in terms of zero-shot performance and reward approximation error. Additionally, we showed that our met hod naturally produces several interpretable components and can be used to visualize where an agent is likely to visit while solving a task. We believe many promising future research directions could be investigated. For instance, SCs could be extended to the hierarchical setting (Machado et al. 2023) by combining high-level SCs predictions and low-level goal-reaching policies. Another promising direction is combining SCs with model-based approaches (Bagot et al., 2023 Alegge et al. 2023). Since model-based techniques excel at short-term planning; they can potentially be used to fine-tune behaviors of SCs agents inside state clusters. We discuss these and other relevant future research ideas in more detail in Appendix
Acknowledgments
This rescarch received funding from the Flemish Government under the "Onderzoeksprogrammaa Artifcile Intelligentie (AI) Vlaanderen" programme. It was also supported by the euROBIN (European ROBotics and AI Network) [No. 101070596). This study was financed in part by the Coordenação de Aperfeigoamento de Pessoal de Nível Superior- Brasil (CAPES) - Finance Code 001.
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (cds.). Advances in Neural Information Processing Systems 94 (NeurIPS 2021), 2021. URL
Lucas N. Alegre, Ana L. C. Bazzan, and Bruno C. da Silva. Optimistic linear support and successor features basis for optimal policy transfer. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 394-413. PMLR, 17-23 Jul 2022. URL wwwreeddiigsmm..... mlr. aaress/v122/alggeeee html
proceedings.
Lucas N. Alegre, Ana L. C. Bazzan, Ann Nowé, and Bruno C. da Silva. Multi-stop generalized policy improvement by leveraging approximate models. In Advances in Neural Information Processing Systems, volume 36, 2023.
19

---

Published in Transactions on Machine Learning Research (07/2025)
Safa Alver and Doina Precup. Constructing good behavior basis for transfer using generalized policy updates. In Proceedings of the Teuth International Conference. on Learning Represenlations, 2022a. URL
https://openreviers hetp:///eee//rrr////////////////
Safa Alver and Doina Precup. Understanding decision-time vs. background planning in model-based rein- forcement learning. arXiv e-prints, art. 2XXi::22060044422 jun 2022b. doi: 404485000/Xiii222.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Picter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advonces in neural information processing systems, 30, 2017.
Louis Bagot, Lynn D'Eer, Steven Latré, Tom De Schepper, and Kevin Mets. Gpi-tree search: algorithms for pecision-time planning with the general policy improvement theorem. In Adaptive and Learning Agents Workshop (ALA), collocated with AAMAS, 29-30 May, 2023, London, UK, pp. 1-8, 2023.
Andre Barreto, Will Dabney, Remi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, Long Beach, CA, USA, 2017a. Curran Associates, Inc.
Andre Barreto, Will Dabney, Remi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hassolt, and David Silver. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems, volume 30, 2017b.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pp. 501-510, PMLR, 2018.
André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Shibl Mourad, David Silver, Doina Precup, et al. The option keyboard: Combining skills reinforcement learning. Advonces in Neural Information Processing Systems, 32, 2019.
André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences, 117(44))30079-30087 2020. ISSN 0027-8424. doi: 10.1073/ 10.1073/pmas.19000
Andrew G. Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrele Event Dynamic Systems, 13(4):341 01933399999911 2003. ISSN 1573-7594. doi: 10.1023/ (100//AA111269111077555
Dimitri P Bertsekas and John N Tsitsiklis. An analysis of stochastic shortest path problems. Mathematics of Operations Research, 16(3):580-58) 595, 1991.
Diana Borsa, André Barreto, John Quan, Daniel J. Mankowitz, Rémi Munos, H. V. Hasselt, D. Silver, and Schaul. Universal successor features approximators. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019. URL https://opeerrviii..eee/////////
Hd=H111JmR5ym
Wilka Carvalho Carvalho, Andre Saraiva, Angelos Filos, Andrew Lampinon, Loic Matthey, Richard L Lewis, Honglak Lee, Satinder Singh, Danilo Jimenez Rezende, and Daniel Zoran. Combining behaviors with the successor features keyboard. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Newral Information Processing Systems, volume 36, pp. 9956-9983, Cur- ran Associates, Inc., 2023a. URL https: /tt.att////pprreeiiins 11529222007044415333330000000002 http://pwwoedddgg.eoor.rr.eeeeep nwwrins.rnncomm http:///ww..ppppp..eeeeee///////
20

---

Published in Transactions on Machine Learning Research (07/2025)
Wilka. Torrico Carvalho, Angelos Filos, Richard Lewis, Honglak Lee, and Satinder Singh. Composing task knowledge with modular successor feature approximators. In Proceedings of The Fleventh Inter- national Conference on Learning Representations, 2023b. URL https::/openrrevier http::/pppearveieeennet/ttorriie net/forunfffr
DrtSxIz40Ib
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Newral Computation, 5(4):613 624, 1993. doi: w..1662/neco, 19t...999.5.4.611.
Benjamin Eysenbach, Abhishek Gupla, Julian Ibarz, and Sergey Levine. Diversily is all you need: Learning skills without a reward function. In Proceedings of the 7th International Conference on Learning Repre- sentations, New Orleans, LA, USA, 2019a. OpenReview.net. URL http::///oparreeeeenneeeeoooooot SJx63jRqFm
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019b.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The information geometry of unsupervised reinforcement learning. In Proceedings of the Tenth International Conference on Learning Representations, 2022. URL https::/ htt.:pparrrreeeeeeeeeorrrrrr.m htte://orommmnddddtu/u
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Marc G Bellemare. Proto-value networks: Scaling representation learning with auxiliary tasks. Proceedings of The Eleventh International Conference on Learning Representations, 2023. URL https: //openreview wew.e//oormmiiiooootttttt.
Francois Fouss, Alain Pirotte, and Marco Sacrens. A novel way of computing similarities between nodes of a graph, with application to collaborative recommendation. In The 2005 IEEE/WIC/ACM International Conference on Web Intelligence (WT005), pp-
TEFF, 2005.
Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Unsupervised zero-shot reinforcement learning via functional reward encodings. arXiv e-prints, art. arXiv:2402.171335 2024.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, pp. 1582-1591, 2018.
Diego Gomez, Michael Bowling, and Marlos C Machado. Proper laplacian representation learning. In The Twelfth International Conference Learning Representations, 2024.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. ICLR Workshop, 2017.
Christopher Grimm, Irina Higgins, André Barreto, Denis Teplyashin, Markus Wulfmeier, Tim Hertweck, Raia Hadsell, and Satinder Singh. Disentangled cumulants help successor representations transfer to new tasks. CoRR, abs/1911.1088 2019. URL httpp:///arxiv.or/aaa//////
10866
S. L. Hakimi. Optimum locations of switching centers and the absolute centers and medians of a graph. Operations Research, 12(3):450 -459, 1964. ISSN 0030-364X. doi: 10.128//prr.12.3..
Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In Proceedings of the 8th International Conference on Learning Representations, 2020.
Yuu Jinnai, Jee Won Park, Marlos C Machado, and George Konidaris. Exploration in reinforcement learning with deep covering options. In International Conference on Learning Representations, 2019.
Leonard Kaufman. Partitioning around medoids (program pam). Finding groups in dala, 344:68 125, 1990.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learn- ing: A review and perspectives. Journal of Artificial Intelligence Research, 75:1401-144 2022.
21

---

Published in Transactions on Machine Learning Research (07/2025)
Romain Laroche and Remi Tachet Des Combes. On the occupancy measure of non-Markovian policies in continuous MDPs. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learn- ing, volume 202 of Proceedings of Machine Learning Research, pp. 18548-18562. PMLR, 23-29 Jul 2023. html
URL httpps://proceednngs www.eng.........................
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: Unsupervised reinforcement learning benchmark. Tn Proceedings of the Thirly- fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL htttp:://pperrevee net/Eorummiddllw<kop
Lucas Lehnert, Michael L. Littman, and Michael J. Frank. Reward-predictive representations generalize across tasks in reinforcement learning. PLoS Computational Biology, 2020.
Lucas Lehnert, Michael J Frank, and Michael L Littman. Reward-predictive clustering. arXiv preprint arXiu:2211.0398/ 2022.
Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28(2):129-137, 1982.
Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. In International Conference on Learning Representations, 2018.
Marlos C. Machado, Andre Barreto, and Doina Precup. Temporal abstraction in reinforcement learning with the successor representation. Journal of Machine Learning Research (JMLR), 24(80): 2023.
Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in applied probability, 29(2):429 443, 1997.
Rahul Ramesh, Manan Tomar, and Balaraman Ravindran. Successor options: An option discovery frame- work for reinforcement learning. In Proceedings of the Twenty-Eighth International Joint Conference Artificial Intelligence, 2019.
Chris Reinke and Xavier Alameda-Pinedas Successor feature representations. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https::/ wttpaa/wwwww.aaeeeeeeooootaattta
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning (ICML), volume 37 of Proceedings of Machine Learning Research, pp. 1312- Tille, France, 07-09 Jul 2015. PMLR.
Alexander A. Sherstov and Peter Stone. Function approximation via tile coding: Automating parameter choice. In J.D. Zucker and I. Saitta (ods.), SARA 2005, volume 3607 of Lecture Notes in Artificial Intelligence, pp. 194-205. Springer, Berlin, 2005.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. The MIT Press, second edition, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. et al. Deepmind control suite. arXiv preprint w:Xiv:1801.00600 2018.
Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volune 34, pp. 13 23. Curran Associates, Inc., 2021.
Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In Proceedings of The Eleventh International Conference on Learning Representations, 2023. URL httts:://ppenrer/ net/forum?idd-Y EcC_
22

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Srinivas Venkattaramanijam, Eric Crawford, Thang Doan, and Doina Precup. Self-supervised learning of
distance functions for goal-conditioned reinforcement learning. arXiv preprint or arxi:: 19:::1907.00000 2019.
Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
Kaixin Wang, Kuangqi Zhou, Qixin Zhang, Jie Shao, Bryan Hooi, and Jiashi Feng. Towards better laplacian representation in reinforcement learning with generalized graph drawing. In International Conference Machine Learning, pp. 11003-1101 PMT.R, 2021.
Yifan Wu, George Tucker, and Ofir Nachum. The laplacian in RL: Learning representations with efficient approximations. In Proceedings of the Seventh International Conference on Learning Represenlations, 2019. URL https://openreeviet http://wwwee..etteoroomooocooaoa
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint Xr1:222001.1:: 19425, 2022.
Tom Zabavy, Andre Barreto, Daniel J Mankowitz, Shaobo Hou, Brendan O'Donoghue, lurii Kemaev, and Satinder Singh. Discovering a set of policies for the worst case reward. In Proceedings of the Ith Interna- lional Conference on Learning Representations, 2021.
A Theoretical Results
In this appendix, we show the proofs of the theorems introduced in this paper, as well as some additional thcoretical results.
A.1 Partitioning for Reward Coverage Proof of Theorem
Theorem 1 = (C.) be partitioning of the state space and let I K-tp the space of K -ippschhitz = reward functions for some distance metric 5 over S. Let & (s) [1c, (s), (s)] be the feature vector indicating subset occupancy: ll., (s) = if E C. and 0 otherwise. Then, reward coverage marimization is bounded as follows:
P(Txa
min weRd SES
Σ (r - (s) w)2
K
5 5 v). a,BEC
(12)
Proof. We are looking for & S Rd such that Ep- min Eses (r (8) - Φ (8) w)²] is minimal,
with r: S + R some state-conditionel reward function and w E IR some weight vector solving the linear least-squares problem for given r and Φ. Let & (s) = [L,Ec,, llEccll be the (onc-hot) indicator vector encoding the location of a state within a partition {C,} of S. Note that C, = S and C. nC, = Vi,j. by definition of a partition. Under this choice of db we can solve the linear problem in w analytically:
Σ (r(ss) (8) w)
Σ
Σ
(r =w;)²
To minimize over w, we can cancel the derivative w.r.t wy and solving for wy:
0 d
Σ (x - w;)² = Σ
(r (x) - w.)) 0
Owi
Ow.
=EC
zEC;
Σ (x w:) = 0
Σ r (x)
- w/ = C
DEC,
23

---

Published in Transactions on Machine Learning Research (07 (07/)025)
So the optimal wi corresponds to the average reward on cluster C₁ and the whole w discretizes the reward function on the state space S. Tet be K-Lipschitz with respect to d in the state space and the squared distance in the output space, i.e. V2 E S
(s1) r (s2)) & Kō s2).
We know from the prior computation of the optimal w that for x E C₁ d (α) w |oT
1
Evec, (v). Then,
2
d
(8) w)
Σ Σ
r(x)
Σ r (v)
C
SSS
f=1 xEC,
VEC:
Noting that vz and constant + 0,r(z) = cr(x)//, so r (z)
Σ
(z),
|C,l
yEC;
2
Σ IYEC
(r(x) -r (r())
|C/ 2
i=1 DcC,
(Jensen's ineq.)
|c,| Σ (r(x) r (1))²
Cil
i=1
=EC,
yEC.
(Lip. prop.) < K
8 (z,y).
E.yEC:
A.2 Partitioning for Unsupervised Skill Discovery
A.2.1 Problem Formulation
In the previous sections, we motivated the use of clustering of the state space to tackle the problem of reward coverage maximization (Section 3.1j. In this section, we further motivate this choice by showing that this clustering is a theoretically complete method for Unsupervised Skill Discovery (USD) 5
Building a strong reward-aguostic skill set to use for down-stream tasks is the core challenge of USD. While the goal of USD is intuitive, its actual objective was only really formalized within a geometric study by Eysenbach et al. (2022) the verlez discovery problem. By considering the discounted state occupancy distribution of any policy n, we can reason over the set of achievable occupancies. Eysenbach et al. (2022) shows that (i) all convex combinations of achievable occupancies are also achievable, therefore the achievable set is polytope in the space of state marginal distributions, and (ti) for any reward function, there is a policy at the vertex of the polytope that maximizes this reward.
Therefore, it is enough to look for the vertices of the polytope; and the USD objective is framed as follows: "gvenn a Controlled Morkov Process, find the smallest set of policies such that every verter of the state marginal polytope contains al. least one policy".
Alegre et al. (2022) have shown that the vertex discovery problem is equivalent to finding the Convex Coverage Set (CCS) for state occupancy measures:
CCS = {00 33 s.L. Vp" p² w 2 pr w)
The CCS corresponds to the set of wwnnooooooiii moon-doinated policies over any given reward function w = [rw (s1),...J" see the paper for further details. Critically, [llegr et al. (2022) further showed that the Successor Represen- tation T corresponds to p", and provide an algorithm to learn the CCS with Successor Features, therefore
5Te terms "skill" and "policy" are used interchangeably.
24

---

Published in Transactions on Machine Learning Research (07 (07/2025)
giving solution to the vertex discovery problem. If can bind the state occupancy measure to its cluster approximation, then we can guarantee that the CCS we learn with Successor Clusters will get closer to optimal as the number of clusters grow.
A.2.2 Theorem and Proof
Let be random state-marginal of the polytope, although the result we will show works on any partitioning over distribution. Let = ]Ci} be set such that Ug C. = S and n. C; 0, i.e,, ş is partition of with clusters C, Define the cluster pseudo-marginal g (<) = c. Enec, P (s:), a discrete approximation of p under clustering Note that since
Ee(s) = EΣ 2= -
Σ ((s))
Cj
1 SEC,
aEC,
B.ECt
Σ Σ P (s:)= =1
&
and 2 0, can be seen as a distribution, which we call a cluster marginl. We can therefore compute the difference between a state marginal and its associated polytope pseudo- marginal:
Theorem 2. Let C be a partitioning of the state space S. Let E A be a state-marginal and define ils associated cluster-marginal as e (s; -)= T.l Em.o,, p (xx) for 9i E Cz. Then, the Wasserstein distance W belween murginals p and under metric 5 is bownded follows:
W8 (p.0) S Σ |C,]
8 (8) SiiDEC;
Σ
(13)
We again recognize a partitioning problem on the right hand side; hence a clustering minimizes the distance between stage-marginals and cluster state-marginals.
other words, considering cluster visitations instead of state visitations when building the CCS will lead to a skill set closer and closer optimal as the number of clusters and their quality increase. Note that this theorem extends Proposition I
Proof. We will compute the Wasserstein distance between a random marginal p and its discrete approxima- tion over the clustcring, p. The Wassertstein distance (Villani et al. 2009) between distributions P and Q. defined in metric space (X, corresponds to the minimum cost of transporting P into Q:
inf / xEA(P.0) xxxx W. (P.Q)
8 (2, X (x, v) dady
where (P,Q) is the set of all couplings of P and Q. The Mongo-Kantorovin duality shows that the Wasserstein has a dual form Müller
1997h:
Ws (P. Q) sup
Eg (x) Ey (v)l
SEL-Lip
where 1-Lip is the of 1-Lipschitz functions under metric 8, (x) (v)) < 8 (x, Under this dual form, compute the difference between the true state marginal p and its discrete approximation g:
25

---

Published in Transactions on Machine Learning Research (07 (07/2025)
W (p:
sup f:1-Lip
Σ -
SES
Σ Σ (8))
Σ
sup f:1-Lip
-
Cil
aEEC;
+EE;
ZiEC;
sup f:1-Lip
Σ Σ ((s)) - -o)
-E p(z;)
Σ
(s;)
[3p
SEEC,
IcEC;
S:EC;
ΣΣ (s:) fil
< sup f:I-Lip
sEEC
Since f is 1-Lipschitz, i.c., If (s1) (s2) $ (81, 82):
(s:) - 11 1 (6i)
Σ (xi) EiEC:
$
If (s:) f (x:))
C
8 23),
C
Therefore, finally
W3 (p. 2) S
8 (s1) Sc,t,E;;
Σ
Cip
B Future Work
Learning USFAs. The challenge of accurately learning a USFA vp (s, a; w) is arguably more complex than that of learning the features indeed, the network needs to be able learn and generalize optimal policies over any reward function encoded via weight vector As result, the statc-of-the-art performance reported in the URL Benchmark [Laskin et al. 2021 for USFA-based agents is still far from the performance of agents trained to be specialized in each task separately. First, we note that selecting the task w that an agent will train next can be seen as a form of curriculum learning. In the benchmark we employed in our experiments, the task vectors w the agent trains on are uniformly randomly sampled, Instead, we can employ existing methods to identify the best next. task w to train an agent on (Alver & Precup, 2022b Alegre al. 2022) to increase the sample efficiency of our method. Second, in the URBL benchmark, agents directly follow and learn from the greedy policy (a 8; w) As arg max, & (s, a; w) w. Instead, we exploit General Policy Improvement (Barreto et al. 2017a GPI) and re-use previously learned weight vectors when solving new task w [Borsa et al.) 2019). in both training and testing phases. Third, there is room for improving the capability of USFAs to generalize across different lasks w. Current methods may not scale to practical applications with large fighddimensional task spaces. To address this, we note that often multiple similar weight vectors w lead to exactly the same policies (c.g., when adding more positive weights in rewards along the optimal trajectory). If future work could foster these symmetrics, USFA learning could be significantly accelerated.
26

---

Published in Transactions on Machine Learning Research (07 (07/)025)
Hierarchical RL. SFs have also been used to build strong and adaptive controller for Hierarchical RL. (Machado et al.) 2023). In this context, the agent drives the online exploration of the environment while building a behavior basis through SFs. For instance, in the Option Keyboard (Barreto et al. 2019) framework, hierarchical meta-policy can be learned to output task vectors, <(w s), which are given as input to the low-level policy in order to trigger precise behaviors. Such a meta-agent can, therefore, abstract out the fine-grained controls to focus on higher-level temporal reasoning. We believe these ideas strongly align with SCs, as the reasoning of the meta-agent can be visualized through the sequence of task vectors it produces. We also note that there is already natural hierarchical structure in the SCs framework presented in this paper. Indeed, we have observed that the USFA ab (8, a; w) can induce policies that traverse all state clusters but cannot fine-tune their position within a cluster. However, we have also trained a goal-reaching policy via Q (s, a; g), able to navigate at shorter time-frames. It is, therefore, straight forward to activate the goal-reaching policy to the highest-rewarding state g = arg max r (s) once the highest-valued cluster C; reached. This enables the agent perform within-cluster movement and addresscs one of the most prominent limitations of our method. We demonstrate this in Appendix [C.] It might also be possible to activate the goal-reaching policy at different times during the trajectory, leave the combination of SCs and hierarchical techniques promising future research direction.
Clustering methods. In the previous section, we discussed the problem of the sampling distribution induced by the replay buffer in an offline RL setting. This leads to a discrepancy between Theorem I for- mulated as a sum/integral over the state space, and the sampling distribution we have access to. Algorithms like K-Means can possibly over-sample regions following the sampling distribution. We argue that a better clustering objective is to minimize the radius of the cluster, to ensure all regions cover around the same amount of time-steps. This objective is sometimes called Metric K-Centers (Hakimi) 1964) and has good approximate solutions. However, this objective itself can be susceptible to outliers and hard to compute online; further study on clustering algorithms is required. With the right approach, a non-linear clustering algorithm like DBSCAN could be a promising alternative. In our experiments, we use fixed clusters of the farthest-first traversal algorithm (K-Centers) in the Maze domain and an Online K-Means algorithm for the robotics domains for the sake of online performance comparison.
Model-based R.L and SCs. As briefly mentioned Section 1 model-based RL (MBRI) could, in theory, solve zero-shot RL provided a perfect model and infinite search depth. This is, of course, unrealistic, as model errors compound for deeper searches and longer horizons. However, the potential of MBRL for zero-shot RL remains, especially for SCs. As have discussed, SCs allow to perform large-scale movement in the MDP, navigating through potentially hundreds of regions. However, they are unable to fine-tune their position within the clusters. In contrast, model-based methods excel at short-term planning and optimization before compounding errors become too big of an issue. We therefore believe they could complement each other greatly, where the long-term SC approximation can be used for bootstrapping at the leaves of a search of depth k. Several frameworks have been proposed for combining SFs, learning models, and planning (Bagot
et al. 2023 Alegre et al.) )023).
We believe all of the future research ideas discussed above could yield significant improvements to the field of zero-shot RL, which is critical to moving toward truly general agents.
C Experiments Details
C.1 Hyperparameters
The hyperparameter used in this paper were mostly taken from Touati et al. (0023) Table summarizes them, with a section for hyperparameters introduced in this paper.
The architecture and training algorithms (TD3) for the USEA and goal-conditioned agent were also taken from Touati et al. (2023). 2 in the following quote refers to w E Rd for the USFA and g E S for the goal- conditioned agent: "we first pre-process (s.a) a) and (s,z) separately by two feedforward networks with two hidden layers (each with 1024 units) to 512-dimentional space. Then we concatenate their two outputs and
27

---

Published in Transactions on Machine Learning Research (07/2025)
Table 1: Hyperparameters used in the Experiments (Section
Hyperparameters from Touati et al. (2023)
Value
Replay buffer size
5 x 10 (10 x 10t for maze)
Representation dimension d
50 (100 for maze)
Batch size
1024
Discount factor Y
0.98 (0.99 for maze)
Optimizer
Adam
Learning rate
10 4
Mixing ratio for sampling Momentum cocfficient for target networks
0.5
0.99
Stddev for policy smoothing
0.2
Truncation level for policy smoothing
0.3
Number of gradient steps
106
Number of reward labels for task inference
10*
Reguralizalee weight for orthonormality loss (Taplacian)
1
Hyperparameters from this paper
Value
Latent space dimension
50
Non-stationary K-Means learning rate
0.1
pass it into another 2-layer feedforward network (each with 1024 units) to output a d-dimensional vector". The policy network is the same, with the output space being the action space, applying Tanh activation since the actions are bounded in [-,,
A goal-conditioned agent was used in Touati et al. 20233 as well for the sake of zero-shot comparison. Similarly to their approach, we use the ratio hyperparameter to sample 50% of the goals as the next state in the transition and 50% as other random states in the batch (following Hindsight Experience Replay ideas). In order to accelerate the learning of the goal-conditioned agent, we used a pessimistic initialization 11((1 7)) 7) to avoid longer convergence times due the overestimation bias.
C.2 Reward Functions used in the Experiments (DeepMind Control Suite)
Figure 14 we include descriptions of the envirouments and reward functions introduced by Touati et al.)
2003 which were used in the experiments presented in Section 5.1
C.3
Comparison with the -awaad--ackkwrd Representation Method
In Figure 15
we extend the results shown in Figure by including the Baaaard-da (FB) representation method Touati et al. 2023). Although our primary goal design reward features that improve upon Successor Foature-based tochniques, our method also achieves competitive results- outperfoiun FB in two out of three robotic tasks. FB state-of-the-arl technique for unsupervised zero-shot, reinforcement learning that does not fall within the class of methods we aim to improve upon, as discussed at the end of Scction 5.1
C.4 Order for Learning Cluster Features
When learning the goal-reacher and clusters, it is most intuitive to train all clements in order first Q (s, a: g): then to (s), and finally {C.)²= to generate = [1c (s), (s)). However, the benchmarking study of Touati al. (2023) trains the reward features within single loop. For fairness of comparison, we also train
6 Replace by the space dimension instead of for goal-conditionel agent.
28

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Point-mass Maze: a 2-dimensional continuous maze with four rooms. The states are 4-dimensional vectors consisting of positions and velocities of the point mass (x, y, Ma, and the actions are 2-dimensional vectors, At test, we assess the performance of the agents on 20 goal-reaching tasks (5 goals in each room described by their (x., y) coordinates).
Walker: planar walker. States are 24-dimensional vectors consisting of positions and velocities of robot joints, and actions are 6-dimensional vectors. We consider 4 different tasks at test time: walker__tt and reward is a combination
terms encouraging an upright torso and some minimal torso height, while wa kerr wa k and wa ker_run run rewards include component encouraging some minimal forward velocity. wa ker_flip reward includes component encouraging some mininal angular momentum.
Cheetah: running planar biped. States are 17-dimensional vectors consisting of positions and velocities of robol joints, and actions 6-dimensional vectors. We consider different tasks at test time: cheet ah walk and cheetah run rewards are linearly proportional to the forward velecity up to some desired values: 2 m/s for walk and 10 m/s for un. Sim- ilarly, ker_wa "__backwasss reaching some minimal backward velocities.
and ker_run_bactee bww.baaaaaarooooooo rewards encourage
Quadruped: four-leg ant navigating in 3D space. States and actions are 78- dimensional and 12-dimensional vectors, respectively. We consider 4 tasks test time: gpadruped_start reward encourages an upright torso. quadruped wa and quadr uped. run include term encouraging some minimal torso velecities. quadruped includes encouraging some minimal height of the center of mass.
Figure 14: Description of the environments as provided by Touati et al. 2023 based on the DeepMind
Control Suile TTssa et al. 2018
SC Minpath (ours)
SC Laplacian (ours)
Minpath
SF Laplacian
Quadruped
Walker
Cheetah
800
800
800
Rotarn
10:00
Rotur
6B0
ueturn
600
600
600
Jump
Flip
Backward
Task
Task
Task
Figure 15: Mean returns obtained in the Quadruped, Walker, and Cheetah tasks from the URLB benchmark. We compare two variants of our method, Successor Clusters (SC), shown shades of red, and two Successor Feature (bF)-basod techniques, shown in shades of blue. Both sets of variants are basod on Minpath and Laplacian reward features. The Forward-Backward (FB) method shown in grey.
all elements simultaneously in the quantitative experiments. We use an online and non-stationary version of K-Means that updates the cluster centers with exponential smoothing with weight 0.1.
C.5 Reward Families Employed in Section 5.2
In Figure 16 we display 5 instances of reward functions from each of the reward families generated for the experiments discussed in Section 5.2
C.6 Hierarchically Combining Successor Clusters and Roal-Conditionee RL
While this is beyond the scope of the zero-shot representation we aim to build in this paper, in Section we discuss the possibility to combine the smooth, long-term movement of our SCs USEA agent with the more precise and short-term movement of the goal-conditioned policy. This can be achieved quite casily by using the SCs USFA agent until the highest valued cluster reached, s E C, J arg max, w;. We then switch to the goal-reaching policy: its goal then simply be the highest-valued state within
29

---

Published in Transactions on Machine Learning Research (07/202) 2025)
+
+
+
Figure 16: The 4 reward families chosen for reward prediction evaluation. By row, top to bottom: goal- reaching, trajectories, Gaussians, smooth maps.
the cluster, g arg max r(s). This will fine-tune the position of the agent within the cluster, though it will ignore local reward structure to only reach the highest-value state; arguably this is a reasonable approximation to make locally. We demonstrate this simple improvement on Successor Clusters in Figure 17 on the same reward function as presented in the experiments in Figure [1b We can see that the resulting trajectory reaches the highest-value state in the cluster, while the SCs USEA agent stopped in the border of the highest-valued cluster.
D Code snippets
In this section, we provide simplified code snippets for Successor Clusters: the clustering and minimal path function. This code assumes usage of the URLB repository [Laskin et al. 2021 with the SFs implementation from Touati et al. (2023)
First, we provide clustering wrapper that can be used on top of any FeatureLearner, using it as a latent space learner to provide cluster occupancy reward features (file sf sy)... In the context. of this code, this requires a non-stationary K-Means implementation:
class MnnStat_KMeans:
def __init__ (self, n_clusters, alpha=0.1
self c_clusters n_clusters
sluster_centeree self. alpha alpha
30

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Figure 17: Using the SCs USFA (trajectory red-to-yellow) until the highest-reward cluster is reached, from which point we trigger the goal-rcaching policy to reach the highest-roward state in the cluster (trajectory black-lo-whitt). The background corresponds the true reward function, as Figure 11b
def partisl_ fit (self, data):
N, dim mata.shapp
alpha self.
# initialize centere (naive: randon elements)
if self bluster_centers is None:
self wlwsterr
data[torch. randperm (N) [:self. n_ clusters]]] clone ()
first step: directly update to the new centers Lloyd's algorithm (assign update)
alpha
dista torch. cdist self clascucertrrce.t.eee. ..1 dists new new_cluster_center n_ n_clusters)])
17
torch stack [[ata [i==] mean (0) for range
18
self ceeuutrrrenttrs..... self wwwsterrcenterrsee...eeee ((1-alpha)
19
20
def predict (self, data):
21
dists torch. cdist (data, self cluster_centere.. -, dists. (-1)
22
23
return 1.long()
Listing 1: Simple Non-stationary K-Means.
We can use this to build the cluster occupancy reward features:
class ClusterLearner (FeatureLearner):
**. Featurelearner wrapped around PeatureLearner (that used for latent space).
Breaks down into clusters
phi=[s in cluster ....] **n
def _.init__selll feature_learner, z_dim, device) None: www.feeaeeeerrreeee feature_learner self.z_dim z_dim self.device device
cluster_aggo MonStat_KKeat (1_clusters=z_din)
self.fitted False
10
11
def forwarddseff. obs: torch. Tensor, action: torch. Tensor, next_obs: torch. Tensor, future_obs: phis = folfu.fatuue_learreeeee feature_ net (obs) detach aelf clusterraleeee clutter.llg..paatia...ttt.ttttt. partial,
12
13
14
self feature leareerr action, next_obe, future_obs)
15
def faature_c(see) obs):
phis = follfffeatuee_larreeea feature_net (obs) detach ()
onehot_clusters onon__ho (self cluster algo. ot (phis), nus_claases=selff z_dim)
onehot clusterrleerre.ee (self device) to floath
Listing 2: Cluster Occupancy Reward Features.
31

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Our implementation is based the minimum distance between two states. Assuming provided goal- conditioned agent (GoalReacher) that can estimate the Q-value in Equation
class MinimalPath (Patureaar)((tttueeeeeerere))
... Assuming trained GoalReacher agent, the features approximate the average minimal path from g. n*n
def _.init___ obs_dim, action_dim, z_dim, hidden_dim) None: super (0...iii......... (obs_din, action_dim, z_dim, sidden_dim) self g.f..gaarrecher GoalReacher (obs_dim, action_dim, z_dim, hidden_dim)
def forl,rd obs: torch. Tensor, action: torch. Tensor, next_obs: torch Tensor, future_obe (orch.Tengor).
10
goals next_obe [torch.. randperm (len (next_obs))]
11
# Phi loss
12
latent_obs - self. fulfffeature_nte (obs)
latent_goals self feature
latent_goals) pow (2) sum (-1) dist_pred torch torch. (trrch clamp (didttpred, min-le-6)) dist_q self goalreacher compute_ distance (obs, goals)
dist_pred (latent_obe
latent_loss (dist_q dist_pred) pow (2) (ean latent_loss
Listing 3: Learning Minpath Latent Space from GoalReacher Agent.
The GoalReacher agent can either be trained in tandem with the latent space or at an earlier stage; it can be implemented in many ways, for example offline with the update function below:
class CoalReacher:
*.. A value learns shortest path The actor learns the associated policy (with TD3). **.
g,
of (neg) steps.
def __init__ (self, obs_dim, action_dim, z_dim, bidden_dim) super i___iittt_t___ddeee_ action_dim, z_dim, hidden_dim) self. qnet
obs_dim, action_dim,
fiddendiim/2,, hidden_dinn out out_dim=1, preprocess-True, add_trunk-False) same as USFA
10
self. qnet_ targ ForkardMap abs_dim, action_dim,
11
hidden_dim//2 hidden_dim, out out_din=1,
12
preprocees-True, add_trunk=False) same as USFA self actor - Actor (obs din, obs_din, action_dim,
13
14
aaddendimm/2? hidden_dix. preprocess=True, add_trunk=False) # sane as USFA
1r 1e-4
self actor_aptppp opt torch. optin. Adam (self parattr.arrneeerr(() (), 1r=1r) self self.q_opt. torch optim Adam (self. qnet. parameters (),, Ir=lr)
self apply (utils. veight_init)
23
(self, obs: actor_loss, q_loss self self.com conputo_ (obs, action, next_obs) # Optimize actor self aat.acctot..errooo
Tensor, action: torch. Tensor, next_obs: torch. Tensor):
24
25
28
(set to (ree
27
actor _etor_loss..o backward
28
self sttrtr # Optimize Q update target
29
30
self o_opt. zero_ (ra
31
q.loss. backvardd)
self q_opt. step utils. wolllsssstt_uddeppppeeee
(self quet, self. qnet_targ, 0.01)
def [empute_losses(e1)
obs: torch, Tensor, action: torch. Tensor, next_obs: torch Tensor
future_or obs: _bs:: torch Tensor):
# Hindsight Experience Replay: first 50% of the transitions successful
mid 1024//2
shuffle mix_ratio 0.5
38
stack ([torch. aaboorch.raageee.......... (mid) torch. randpern (nid) mmid]) flatten goals next, [shuffle]
39
32

---

Published in Transactions on Machine Learning Research (07 (07/2025)
41
Actor actions
42
best_ action self (obs, goals, .2) sample
43
sett_next_action
actor goals, .2) (ample detach
44
46
TD3 for Q function learning qagi, qag2 self, qubt goals, action) next_qsgl, next_qsg2 (wt.aaiiintottt..g.....eeee.eeee
48
47
tnat_ttaettaaggeeeeeeeeeee (next_o__ _bb,, goals, best past_next_accttee
48
next_qag aarchtaol (next_oba==goals din=-1). fflat () detach
next_qag2 squeeze () detach
49
done = torch. all
50
q_bell -1 0.99 * next_qag (1-done)
51
9_lossi = (q5gl squeaze(] () q_bell q_bell.detace. detach ()) -oo (2) mean ()
q_loss2 = (qsg2. squeaze q_bell detach ()) pow (2) mean ()
58
# Actor loss
best_ 1 best_qsg2 self quet (obe, goals, best best_action. best_qag torch. (best_ qag1 squeeze (), aggee. loas bestt_qsg mean
actor_lose loss, q_loss1 9_loss2
def conpate_distance (self, obs, goals): 61
62
best_action_sg self actoo goals, .2) sample dist_q_ag self quet (obs, goals, best_action_egs detach best_action_ga actor dist_q-gs = self quet (goals, obs, best_action_ga) detach () return -(dist_q_sg + dist_q.ga)/2
63
64
obs, .2) sanple (clip=.3) 33
65
66
Listing 4: Training a GoalReacher Agent.
33


Reviews:
[{"review_id": "6bp0tp4aEf", "reviewer": "Reviewer_Mwrx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n\n- The idea of using Eq. (5), i.e., the minimum expected number of stages between different states, for constructing features for the state space is interesting.\n\n## Weaknesses:\n\nOverall, I find the contributions of the paper quite limited, as \"successor clusters\" means just proposing a new heuristic approach for computing the features of the state space, and this heuristic approach does not convince me very much, because (1) there are some problems in the theoretical results, and (2) the experiments compare do not compare with enough baselines.\nIn particular:\n\n- In Fig. 1 symbols $Q(s,a;g)$ and $\\psi(s,a;w)$ are undefined. I suggest to provide in the caption a note to explain their meaning.\n- At beginning of Section 3, symbol $\\mathcal{W}$ is undefined.\n- In Section 3.1, you say that: \"The main obstacle to solving this problem is that we do not have prior information on the reward functions that an agent may need to solve during transfer. Hence, the representation $\\phi$ of dimension $d$ must linearly cover the widest range of possible reward functions uniformly sampled from the given space of reward functions\". Why is this? I mean, this objective is equivalent to assuming a \"uniform prior\" over rewards in $\\Gamma$. However, note that, since $\\Gamma$ is unbounded, then there is no notion of \"uniform prior\" that can be defined onto it. Simply put, my guess is that if $d<S$, then for whatever choice of $\\phi$ you make, the objective in Eq. (3) is $+\\infty$.\n- What sense does it have to upper bound $1/|C_i|^2$ with $1/|C_i|$ in the proof of theorem 1?\n- In the proof of theorem 1, I think that the passage associated to (triangle ineq.) if wrong. Basically, you are using that $(\\sum_i a_i)^2\\le \\sum_i a_i^2$, but how can you say that?\n- In addition, Theorem 1 is wrong because set $\\Gamma_{K-Lip}$ is unbounded, and so we cannot define a uniform distribution on it.\n- In Fig. 3 there is a error/typo: $\\gamma=1$ cannot be. P.S.: I realized later that you use terminal states. Please, make it clear in the background section adding a proper formalism.\n- The proof of Proposition 1 is missing. However, it is obvious so that it does not require a proof. As such, you should not call it \"proposition\", but use something like \"fact\", or \"observation\".\n- where is $\\pi$ used in Line 7 of Algorithm 1?\n- In Algorithm 1, what is $Q(s,a;g)$? P.s.: found it. Please, make it clearer.\n- What is $\\mathcal{W}$ in Eq. (7)? I mean, how one should set it?\n- You should write \"Euclidean\" and not \"Euclidian\".\n- In Section 4.2, if you have $n$ samples from the reward, how can you compute Eq. (10)? I mean, you do not know $r$ everywhere.\n- In the experiments, why do you not compare with forward-backward representations as in Touati et al. 2023? Why do not you try with other baselines, i.e., other kind of features (beyond those in Touati et al. 2023)?\n- I cannot find in the text what reward functions did you use in the experiment Q1. Specifically, I would like to know if you sample the rewards uniformly from the set of lipschitz rewards and what is the Lipschitz constant.\n- I strongly suggest to improve the presentation of section 4, because it is very difficult to understand how Algorithm 1 is implemented (some symbols/concept undefined).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of zero-shot unsupervised RL, where the environment $(\\mathcal{S},\\mathcal{A},\\gamma,p,\\mu)$ keeps constant, but the reward function can be arbitrary, and is revealed only after a reward-free exploration with the environment. To address it, the authors adopt the method of successor features existing in the literature, that is based on learning the successor features $\\psi(s,a;w)$ for a subset of policies \"linear\" in $w$, for any $w$.\n\nIn this context, the main contribution of the authors is proposing a new method for finding the features $\\phi$. Specifically, they propose to clusterize the state space using as distance between states $s,s'$ the minimum expected number of stages for going from $s$ to $s'$ (the average of $s\\to s'$ and $s'\\to s$), and then to use the one-hot features $\\phi(s)$ representing the membership to of state $s$ to each cluster $\\\\{C_i\\\\}_i$.\n\nFinally, the authors present a practical algorithm for unsupervised zero-shot transfer in RL and conduct some experimental validation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please, see the \"Strengths And Weaknesses\" section.\n\nIn particular, the following adjustments are critical:\n\n-  Answer to all my questions :)\n- Avoid defining a uniform prior over an unbounded set (it is not possible)\n- Fix Theorem 1 (if possible)\n- Improve a lot the presentation of Section 4.\n- Conduct experiments with other baselines, specifically with forward-backward representations as in Touati et al. 2023.\n\nOne of my biggest concerns is with the last point. Touati et al. 2023 explain that forward-backward representations work quite good for the task of zero-shot unsupervised RL, so why using a successor feature method? Please, motivate.", "broader_impact_concerns": "/", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1uoH6MMf4r", "reviewer": "Reviewer_67KV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n1. The literature in SF often assume the reward function is linear in terms of successor feature. This paper broadens this concept to lipschitz rewards.\n\n2. The concept of successor cluster is reasonable and a simple and clear idea. The authors explore this idea, and provides interesting properties of this concept.\n\n3. The paper is well-written and easy to follow.\n\n\n**Weakness**\n\n1. Proposition 1 can be improved : How many clusters do we need or what is the dependency on the problem parameters to cover an optimal policy?\n\n2. It is not clear how tight the error bound on (4) is.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The paper proposes a reward coverage maximization problem in the successor feature (SF) learning scheme, which tries to select feature that well-approximates the reward function. The error is upper bounded by some distance measure, and the authors proposes a method that minimizes this upper bound motivated from the SPP.\n\n2. The paper considers a $K$-lipschitz reward functions with respect to the state, guaranteeing smoothness of reward functions as the state changes.\n\n3. The paper considers occupancy measure in terms of the clusters.\n\n4. A distance function is learned to identify the clusters that minimizes the error bound in (4).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The class of reward function requires more context around it. For example Tasfi et al. considers quadratic form of rewards for SF learning.\n\n2. The contribution part can be written in a more concise manner. For example, the last item seems to be merged into the fourth item. \n\n\n3. The reward function seems to be deterministic from each state. Several comments is required when the reward function depends also on the action, and its next state.\n\nTasfi, Norman L., and Miriam Capretz. \"Non-Linear Rewards For Successor Features.\" (2020).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EAi4OESUMI", "reviewer": "Reviewer_Jgad", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- SCs enables agents to adapt to new reward functions without any additional training. The paper empirically demonstrates that SCs outperform state-of-the-art methods in continuous control tasks..\n- The paper provides a strong theoretical foundation for SCs. It introduces a mathematical formulation for learning reward features and proves that SCs allow for arbitrarily good approximation of any Lipschitz reward function. It also shows that as the number and quality of clusters increase, the set of policies induced by SCs converges to a set containing the optimal policy for any new task.\n- SCs offer a high degree of interpretability. Because SCs predict the expected time an agent spends in each state cluster, it's possible to visualize the agent's planned trajectory and understand its behavior. This contrasts with other methods that often rely on less interpretable features.\n- The method allows for controlling the agent's behavior by adjusting the weights associated with different clusters. It also allows for specifying desired trajectories in terms of cluster visits, enabling a form of imitation learning.\n\n\n\n## Weaknesses\n\n- The specific clustering algorithm used (K-Means in the experiments) has its own limitations, such as sensitivity to initialization and the assumption of isotropic clusters. While the paper mentions exploring other clustering algorithms (like DBSCAN or Metric K-Centers) as future work, the current implementation relies on K-Means or its variants.\n\n- The Universal Successor Feature Approximator needs to learn and generalize optimal policies over any reward function, which might be a more complex task than learning the features themselves. This can limit the scalability of the method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Successor Clusters, a new method for unsupervised zero-shot reinforcement learning. It tries to enable an RL agent to adapt to new reward functions without any additional training. This is achieved by clustering the state space using a novel distance metric based on the minimum number of time steps to transition between states. SCs, a variant of Successor Features, predict the expected time a policy spends in each of these state clusters. The paper demonstrates that Successor Clusters can approximate and maximize new reward functions in a zero-shot manner and shows that the method produces interpretable features, allowing visualization of an agent's likely trajectory. Empirical results show that SCs outperform state-of-the-art methods in continuous control benchmarks, achieving better zero-shot performance and lower reward approximation error.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I think this paper is in good shape.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)