$\pi$2vec: Policy Representation with Successor Features
Authors: Gianluca Scarpellini
Ksenia Konyushkova
Claudio Fantacci
Thomas Paine
Yutian Chen
Misha Denil
Venue: ICLR 2024 poster (2024)
Keywords: Policy representation
offline policy selection
robotics
evaluation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper introduces `pi2vec`, a method that represents black box policies as feature vectors by leveraging a pretrained foundation model and successor feature framework, enabling efficient policy evaluation in resource constrained environments.

Abstract: This paper introduces $\pi$2vec, a method for representing black box policies as comparable feature vectors.
Our method combines the strengths of foundation models that serve as generic and powerful state representations and successor features that can model the future occurrence of the states for a policy.
$\pi$2vec represents the behavior of policies by capturing the statistics of the features from a pretrained model with the help of successor feature framework. 
We focus on the offline setting where policies and their representations are trained on a fixed dataset of trajectories.
Finally, we employ linear regression on $\pi$2vec vector representations to predict the performance of held out policies.
The synergy of these techniques results in a method for efficient policy evaluation in resource constrained environments.

Published as a conference paper at ICLR 2024
π 2VEC: POLICY REPRESENTATION WITH SIIHSUCCHR SUCCESSOR FEATURES
Gianluca Scarpellini* f Istituto Italiano di Tecnologia
Ksenia Konyushkova Google DeepMind
Claudio Fantacci
Google DeepMind
Tom Le Paine Google DeepMind
Yutian Chen Google DeepMind
Misha Denil Google DeepMind
Work done during internship Google DeepMind Corresponding author Sanluca.scarpplling @ it..it
ABSTRACT
This paper introduces 2vec,, method for representing black box policies as comparable feature vectors. Our method combines the strengths of foundation models that serve as generic and powerful state representations and successor features that can model the future occurrence of the states for policy. #2vc represents the behaviors of policies by capturing statistics of how the behavior evolves the features from pretrained model, using a successor feature framework. We focus the offline setting where both policies and their representations trained on a fixed dataset of trajectories. Finally, we employ linear regression on #2ve vector representations predict the performance of held out policies. The synergy of these techniques results in a method for efficient policy evaluation resource constrained environments.
INTRODUCTION
Robot time is an important bottleneck in applying reinforcement learning in real life robotics applications. Constraints on robot time have driven progress sim2real, offline reinforcement learning (offline RI.,, and data efficient learning. However, these approaches do not address the problem of policy evaluation which is often time intensive as well. Various proxy metries were introduced to eliminate the need for real robots in the evaluation. For example, in sim2real we measure the performance in simulation (Lee et al. 2021). Evaluation (OPE) methods (Gulcehre al. 2020 Fu et al. 2021). For the purpose of deploying policy in the real world, recent works focused on Offline Policy Selection (OPS), where the goal to select the best performing policy relying only offline data. While these methods useful for determining coarse relative performance of policies, one still needs time on real robot for more reliable estimates (Levine al. 2020
offline RL we rely on Off-policy
Our proposed #2vec aims at making efficient use of the evaluation time. Efficient offline policy evaluation and selection is relevant in reinforcement learning projects, where researchers often face the challenge of validating improvements. 72vec enables researchers to make more informed decisions regarding which new policy iterations prioritize for real-world testing to identify and discard less promising options early in the development process. In particular, we predict the values of unknown policies from set of policies with known values in an offline setting, where a large dataset of historical trajectories from other policies and human demonstrations is provided. The last step requires policies be represented as vectors which are comparable and thus can serve as an input to the objective function. Prior work from Konyushova al. (2021) represents policies by the actions that they take on a set of canonical states, under the assumption that similar actions similar states imply similar behaviour. However, this assumption is sometimes violated in practice. This work aims at finding more suitable representation by characterizing the policies based on how they change the environment.
To represent policies, our method #2vec combines two components: successor features and foundation models. We adapt the framework of Q-learning of successor features (Barreto al. 2017) to the

---

Published as a conference paper at ICLR 2024
Foundation
T
Y
V 22
2
22vec
3
Offline
T
Figure 1: T2vec method relies the successor feature framework, that we adopt in combination with a dataset of offline demonstrations and a visual foundation model Φ. 2vee represents each policy a; as feature vector e E IR.. v encodes the expected behavior of policy when deployed on an agent.
offline setting by applying the Fitted Q evaluation (FQE) algorithm (Le et al. 2019) which is typically
used for off-policy evaluation (OPE). In this work the features for individual states are provided by general purpose pretrained visual foundation model (Bommasani al. 2021). The resulting representations can be used as a drop in replacement for the action-based representation used by Konyushova al. (202I).
Our experiments show that 2vec achieves solid results in different tasks and across different settings. To summarize, our main contributions the following:
We propose 22eec, a novel policy representation of how the policies change the environment, which combines successor features, foundation models, and offline data;
We evaluate our proposal through extensive experiments predicting return values of held out policies in 3 simulated and 2 real environments. Our approach outperforms the baseline and achieves solid results even in challenging real robotic settings and out-of-distribute scenarios; We investigate various feature encoders, ranging from semantic geometrical visual foundation models, to show strengths and weaknesses of various representations for the task at hand.
2 RELATED WORK
Representation of black-box policies. In this paper, our objective is to create vector representations for policies to predict their performance, We treat policies as black-boxer (i.e., no access internal state, parameters, or architectures) that yield actions for given observation. It is important to emphasize that our objective differs from representation learning for RL (Schwarzer et al. 2020) Jaderberg et al. 2016 Laskin al. (2020), we focus representing policies rather than training feature encoders for downstream tasks.
Konyushova al. (2021) studied setting where the goal is identify the best policy from set of policies with a dataset of offline experience and limited access to the environment. Each policy is represented by vector of actions at fixed set of states. While this representation performs well in certain applications, it may not be the most effective for predicting policy performance. For instance, consider two policies that generate random actions at each state. These policies do not exhibit meaningfully different beahviour, so for policy evaluation purposes, we expect them to be similar. However, the action policy representation categorizes these policies as different. This paper proposes method to address this limitation by measuring trajectory- level changes in the environment.
In BCRL CChang al. 2022), a state-action feature representation proposed for estimating policy performance. However. the representation of each policy is independent of other policies and thus cannot be employed to regress the performance new policies given a set of evaluated policies.
Offline Policy Evaluation. Off-policy Evaluation (OPE) aims to evaluate policy given access to trajectories generated by another policy. It has been extensively studied across many domains (Li al. 2010 Theocharous al., 2015; Kalashnikov et al. 2018 Nie et al.) 2019). Broad categories of
OPE methods include methods that use importance sampling (Precup] 2000). binary classification
(Irpan et al. 2019), stationary state distribution (Liu et al. 2018). value functions (Sutton et al. 2016
2

---

Published as a conference paper at ICLR 2024
Le et al. 2019), and learned transition models (Zhang et al. 2021 well as methods that combine two or more approaches (Farajtabar et al. 2018). The main focus of the OPEs approaches is on approximating the return values function for trained policy, while #2vec goes beyond classical OPE and focuses on encoding the behavior of the policy as vectors, in such a way that those vectors are comparable, fit performance predictor.
Foundation Models for Robotics. Foundation models are large. self culf-super models (Bom- masani et al. 202T) known for their adaptability in various tasks (Sharma et al., (023). We compare
three representative foundation models (Radford al., 2021; Dosovitskky
al. 2021] Doersch
et al. 2022). Our proposal, 22vec, is independent of the feature encoder of choice. Better or domain-specific foundation models may improve results but are not the focus of this study.
3
METHODOLOGY
3.1 OVERVIEW
Our setting is the following. We start with large dataset of historical trajectories D, and policy- agnostic state-feature encoder ø : S IR.. Given a policy и, our objective is to use these ingredients to create policy embedding v E RN that represents the behavior of π (and can be used predict its performance).
We aim to create this embedding offline, without rumning the policy in the environment. Although we can evaluate ı for any state in our historical dataset D, we emphasize that we do not have access any policy trajectories from n, which significantly complicates the process of creating embedding that captures the behavior of π
Our method 22ee has three steps:
1. Choose policy-agnosiee featureeature encoder c. We discuss several options for & below and in the experiments; however, 2vee treats the policy-agnostic state-feature encoder as a black box, allowing to leverage generic feature representations in work.
2. Train policy-specific state-feature encoder 4/ (S, A) - R this step we combine the policy-agnostic state-feature encoder Φ), and the policy to create policy-specific state- feature encoder by training on the historical dataset DB The policy-specific state features (s) capture statistics of how would change the environment were it to be run starting from the state s.
3. Aggregate the policy-specific state ftateffettres to create state- aante-agnottc policy features T that represent the behavior of π in inte-inddeendeent way.
Using the steps outlined above we can collect dataset of pplicy-specific state-independent features paired with measured policy performance. This dataset can be used to train model that predicts the performance of a policy from its features using supervised learning. Because we compute features for policy using only offline data, when we receive a new policy we can compute its policy-specific state-independent features and apply the performance model to predict its performance before running it in the environment. In the following sections we expand on each step.
3.2 POLICY-AGNOSTIC STATE FEATURES
The role of the feature encoder is to produce an embedding that represents an individual state of the environment. In this paper focus on state encoders Φ + R that consume single images Generically our method agnostic to the input space of the state-feature encoder, but practically speaking it is convenient work with image encoders because that gives us access to wide range of pretrained generic image encoders that are available in the literature.
We also consider few simple ways to construct more complex features from single image features. When each state provides multiple images we embed each image separately and sum the result to create state embedding. We also consider creating embeddings for transitions (s, s') by computing A 8') A ((s') ((s). Both cases allow to leverage features from pretrained models.
3

---

Published as a conference paper at ICLR 2024
π
Trajectory:
2
2 Φ
Figure 2: Given a trajectory from the dataset of offline demonstrations, we train successor feature (s:) to predict the discounted sum of features Σ (((ts)) where ø is visual feature extractor and a policy. Intuitively, ((s;) represents semantic changes in the current state of the environment 8t, while successor feature 1/ summarizes all future features encoded by ø if actions came from policy n.
3.3 PELIC-SPPECIEL STATE FEATURES
The next step is to use the policy-agnosti agmostic state-feature encoder á that provides a generic represen- tation for individual states to train a policy-specifie state-feature encoder v/ (S,A) - R that represents the effect that π would have on the environment it were run starting from the given state.
The work of Dayan (993);; Barreto al. (2017) successor features provides basis for approach to policy representation. We briefly review successor features here, and comment below on how we make use of them. We refer the reader recent literature covering successor features
Lehnert Littman (2020): Brantley et al. (202T); Reinke & Alameda-Pineda (021))
Suppose that the reward function for task be written linear function
r(s, 0, s') (0(s, a, s'), Wisk),
where d(s, a, s') E RN encodes the state-transition as a feature vector and Wask E RN are weights. Barreto et al. (2017) observe that if the reward can be factored as above, then the state-action-value function for a policy x can be written as
(s, = ((oll))
- 1(8i (1,t))) = -1 a), Wrask),
(2)
where
(s, Biss s)
(s|s') D is transition from the environment, and y is the discount factor. The corresponding state-value function is (s)) Q* (s)) = (v (s, (s)), Winx) & (ve((s) Wis). We will use notation (s) A v (+)) frequently throughout the remainder of the paper.
The value of (x) is known as the prrcessorfeature of the state s under the policy Successor features were originally motivated through the above derivation as way of factoring the value function of policy into lask-independent behavior component (the successor features) that is independent of the task, and a task-dependent reward component that independent of behavior.
For our purposes will mostly ignore the reward component (although we return to it in one of the experiments) and focus on the behavior term shown in Equation S This term is interesting to us for two reasons. First, represents the behavior π as future discounted sum of state features along trajectory obtained by running æ beginning from the state In other words, 2 represents the behavior of in terms of
see by inspection of the RHS that the value of vi (s) (s, (s))
4

---

Published as a conference paper at ICLR 2024
the features of the states that will encounter, where the state features themselves given by the policy-agnostic state-feature encoder from the previous section. Figure summarizes the relationship between successor features & and state encoders d.
Second, Equation 3 the Bellman equation meaning that the function v/ (s, a) can be estimated from off-policy data in a task-agnostic way using a modified version of Q-learning, where the scalar value reward in ordinary Q-learning is replaced with the vector valued transition features ((s, a, s'). We rely on Fitted Q Evaluation (FQE, [e et al. (2019), an offline Q-learning based algorithm, and thus, we obtain a representation of policy behavior purely from data without executing the policy in the environment. Given a dataset D and policy æ. FQE estimates its faae--cciionvallee function Q" (s, a) according to the following bootstrap loss:
T((0) ((s.....s'')DDaaaaa [llv% (8, (,(8, + (3()s
(4)
FQE is simple to implement and it performs competitively with other OPE algorithms in a variety of settings (Fu al., (02T) including simulated and real robotics domains (Paine al. 2020
Konyushova et al. (021). We use FQE with our historical dataset ID to train a policy-specific
state-action- network v (i, a), which we then use the policy-specific state-feature encoder ei (s, x (s)) by plugging in the policy action.
3.4 STATE- POLICY FEATURES
We obtain single representation T of a policy æ from the state-dependen successor features (s) for that policy by averaging the successor features over a set of canonical states:
[
(5)
where Dean is set of states sampled from historical trajectories. We sample the canonical states set Dean C ID uniformly from from our historical dataset, as in Konyushova et al. (2021), ensuring that each canonical state comes from a different trajectory for better coverage. We average successor features over the same set Dran for every policy. The intuition behind this representation is that v (s) represents the expected change that ग induces the environment by starting the state s; by averaging over Dean. U represents an aggregated average effect of the behavior of I.
3.5 PERFORMANCE PREDICTION
We aim at predicting the performance of novel, unseen policies. We begin with a dataset of his- torical policies for which have measured performance II {.... (T,, ........ For each pol- icy in this dataset we create an embedding using the above procedure to obtain a new dataset {..., (we m.. RJ and then train a performance model fG = f( nt using supervised learning. Given a new policy т. can then predict its performance before running in the environment by computing the m2vec features for the new policy using the above procedure and applying the performance model to obtain R. = f(
4 EXPERIMENTAL SETUP
In this section we describe the feature encoders, domains, and evaluation procedures, followed by details about our baselines. More details about architecture, domains, and training procedure can be found in the Appendix.
Feature encoder. Firstly, the Random feature encoder employs randomly-initialzz ResNet- 50 (He et al.] 2016). Random features are trivial to implement, and achieve surprisingly strong performance many settings (Rahimi & Recht 2007). Here they serve simple baseline.
Next, we explore with CLIP (Radford et al. 2021). CLIP-network is trained to match image and text embeddings large-scale dataset of image caption pairs. Intuitively, by aligning image and text features, CLIP network is trained to encode high-level semantic information.
Visual Transformers (VIT) (Dosovitskiy et al. 2021 treat images as ID sequence of patches and learn visual features via an attention mechanism. In our experiments the visual transformer is pre-trained on imagenet classification.
5

---

Published as a conference paper at ICLR 2024
Right-door Opan (left)
Assembly (left)
Microwave (top)
Bin picking (left)
Lighl-on (right)
Bution press (left)
Knob-on (right)
Drawer open
(left)
(iii)
(iv)
(v)
Figure 3: We adopt 5 environments. (i) Kitchen: tasks (Knob-on, Left door open, light on, microwave open, and right door open) and 3 points of views. (ii) Metaworld: tasks (assembly, button press, bin picking, and drawer open) and 3 points of views. (iii) Insert gear in simulation (iii) and (iv) on a real robot. (v) RGB stacking on a real robot.
Lastly, we explore pryii- (TAP) (Doersch al. 2022). general- purppss network for point tracking in videos. The network is pre-trained to track arbitrary points over video sequences and as result learns to understand the low-level geometric features in a scene. We use an attention layer trained to select task-relevant features from the TAP model to reduce dimensionality.
This set of feature encoders spans spectrum of properties as they are created by optimising different objectives. At one extreme CLIP features trained to align image features with text description, and encode the semantics of the image. At the other extreme TAP features are trained to track points in videos, and capture low level geometric and texture information. VIT features are in the middle, they need to encode both semantics and local texture to accomplish classification tasks. Depending on the environment and task hand, better state representation is likely to result in better prediction properties of 72vec. We leave the question of finding the best representation as future work.
Domains. We present extensive experiments to support #2vec's capabilities across three simulated domains- Insaert Gear (Sim), Metaworld, and Franka- Kitcha,,, and two real (Real) and RGB Stacking (FigureB). In each domain we use a dataset of offline human demonstrations (Metaworld and Kitchen) and held out policies trajectories (RGBStacking and Insert Gear) for training policy representations. Each policy is treated as black-box where we do not have any prior knowledge about the architecture or training parameters. We provide further details in Supplementary.
Insert Gear
Evaluation. We assess the quality of the policy representations by measuring the ability of the model f to predict the performance of held out policies (see Section 3.5). We adopt k-fold cross validation over the set and report results averaged over cross validation folds. Following previous works offline policy evaluation (Paine al., 2020 Fu al., (02T). adopt the following three complementary metrics. We report further details in the Suppiementary.
Normalized Mean Absolute Error (NMAE) measures the accuracy of the prediction w.r.t. the ground-truth. We adopt MAE instead of MSE to be robust to outliers and we normalize the error to be range between the return values for each environment. Lower is better.
Rank Correlation measures how the estimated values correlate with the ground-truth. Correlation focuses on how many evaluations on the robot are required find the best policy. Higher is better. Regret@1 @1 measures the performance difference between the best policy and the predicted best policy, normalized w.r.t. the range of returns values for each environment. Lower better.
Correlation and Regret@) @1 are the most relevant metric for evaluating m2vec on OPS. On the other hand, NMAE refers to the accuracy of the estimated return value and is suited for OPE.
Baselines.
The problem this paper is to represent policies in such a way that the representations
be used to predict the performance of other policies given the performance of subset of policies. Importantly, to address this problem the representation should 1) encode the behavior of the policy, 2) in way that is comparable with the representations of other policies, and 3) does not require online

---

Published as a conference paper at ICLR 2024
Table 1:
We compare T2vec and Actions representations for Insert-gear (real) and Insert-gear (sim) tasks, as well as for the RGB stacking environment. The table shows the performance and confidence intervals for different feature representations and encoders.
Representation
NMAE 4
Correlation 1
Regret @
RGB Stacking
Actions
0.261 =0.045
0.785 #0.177
0.074 #0.083
VIT
0.224 20.063
0.775 10.146
0.036 10.116
AVIT
0.344 =0.050
0,030
+0.332
0.375
20.206
CLIP
0.330 =0.042
0.342 20.293
0.325 18.180
ACLIP
0.287 20.048
0.583 10.126
0.079 10.126
Random
0.304 =0.006
0.330 +0.334
0.226 10.177
ARandom
0.325 10.109
0.352 +0.348
0.190 10.180
Insert gear (real)
Actions
0.252 =0.028
-0,545 18.185
0.578
148
Random
0.275 =0.027
-0.207 +0.267
0.360
10.162
CLIP
0.198 =0.030
0.618
+0.136
0.267
10.131
ACLIP
0.253 20.228
-0.109 10.100
0.429
10.100
Insert gear (sim)
Actions
0.174 10.015
0.650 1..056
0.427
10.172
Random
0.215 20.026
0.555 10.104
0,422 10.143
TAP
0.164 =0.022
0.680
10.095
0.359 10.184
VIT
0.224 =0.025
0.402 12.199
0.448 10.155
AVIT
0.255 00.024
0.218 13.
0.457
153
CLIP
0.180 =0.031
0.502
+0 0.068
0.298
10.126
ACLIP
0.189 =0.020
0.586 +0.077
0.314 10.117
data. Active Offline Policy Selection (AOPS) (Konyushova et al., 2021) stands alone as a notable work that delves into policy representation from offline data with the task of deciding which policies should be evaluated in priority to gain the most information about the system. AOPS showed that representing policies according to its algorithm leads to faster identification of the best policy. In AOPS's representation, which we call "Actioos", policies represented through the actions that the policies take on a fixed set of canonical states. We build Actions representation as follows. We run each policy on the set of states Dean sampled from historical trajectories. Next, concatenate the into a vector.
resulting set of actions (s))
To the best of our knowledge, the Actions representation is the only applicable baseline in the setting that we adopt in this paper. Nevertheless, OPE methods that estimate policy performance from a fixed offline dataset are standard methodology in offline RL literature. Although these methods do not take the full advantage of the problem setting in this paper (the performance of some of the policies known) they can still serve for comparison. In this paper, we compared against FQE which is a recommended OPE method that strikes good balance between performance (it is among the top methods) and complexity (it does not require a world model) (Fu et al. (02T).
5 RESULTS
We report results for various feature encoders for Insert gear (sim and real) and RGBStacking. Similarly, we report averaged results for over 4 tasks and 3 point of view for Metaworld and over 5 tasks and point of view for Kitchen. Along with results for each feature encoder, we report the average results of picking the best feature encoder for each task (BEST-0). Similarly, we report as BEST-CLIP and BEST-VIT the average results when adopting the best feature encoder between CLIP/VIT and ACLIP/AVIT. We identify the best feature encoder for a task by conducting cross-validation on previously evaluated policies and pick the best encoder in terms of regret I.
Our results demonstrate that (i) #2vec outperforms the Actions baseline models consistently across real and simulated robotics environments and multiple tasks, showcasing the framework's 's effective- demonstrate the applicability real-world robotic
ness in representing policies. Furthermore, settings, specifically in the challenging Insert Gear (Real) environment, where even underperform- ing policies contribute improved policy evaluation. We show that choosing the best model as
7

---

Published as a conference paper at ICLR 2024
Table 2: We evaluate 72vec on Metaworld and Kitchen. The results are averaged over all settings and confidence intervals are reported. BEST-6 is 22ee average performance assuming that we adopt the best & in terms of regret@1 for each task-POV setting. Similarly, BEST-CLIP and BEST-VIT are the best feature encoder between CLIP/VIT and ACLIP/AVIT.
Representation
NMAE 1
Correlation 1
Regret
Metaworld
Actions
0.424 =0.058
0.347 +0.152
0.232 #0.078
CLIP
0.340 =0.035
0.254 10.143
0.250 #0.076
ACLIP
0.325 =0.092
0.286 10.154
0.232 10.086
BEST-CLIP
0.309 =0.027
0.351 10.130
0.194 10.076
VIT
0.303 =0.030
0,280 #0
0.263 #0.091
AVIT
0.315 =0.026
0.162 16.
0.325 20.084
BEST-VIT
0.298 20.029
0,300 10.147
0.244 00.092
Random
0.366 =0.086
0.043 10.150
0.375 10.108
BEST-6
0.289 =0.018
0.460 +0.099
0.153 #0.000
Kitchen
Actions
0.857 10.128
0.326 10.128
0.221 10.080
CLIP
0.417 10.032
0.021 1.0.119
0.317 10.081
ACLIP
0.352 20.02
0.260 10.216
0.244 1.0.81
BEST-CLIP
0.333 =0.025
0.346 20.
0.197 #0.076
VIT
0.385 =0.030
0.030 20.244
0.322 00.00
AVIT
0.344 =0.025
0.155 +0,234
0.251 20.082
BEST-VIT
0.321 =0.024
0.412 40.228
0.151 20.068
Random
0.382 =0.033
-0.017 10.225
0.334 000000
BEST-6
0.392 00053
0.591 10.203
0.070 00.045
a feature-extractor greatly improves results (ii). Finally, we adopt #2vec to solve Equation 2 and estimate policies' return values in the Metaworld's assembly environment, without relying any ground-trun data (iii). Although the successor feature assumption of linearity of rewards is violated, T2vec still ranks policies competitively in the offline setting when compared FQE. In the Appendix, we provide an intuition for choosing the best ø based on the correlation between task difficulty (iv), and we study the effect of different dataset types, such demonstrations and trajectories from held out policies (v), We investigate T2vec's generalization capabilities (vi). including out-of oiffo--iistribution scenarios (vii). We also demonstrate that 22ee represents random policies close in the feature space (viii), and that π 2vec is robust to canonical state coverage (ix) and effective with online data (x).
(i) n 2vec consistently outperforms Actions. We compare T2vec and Actions across all scenarios. Our method outperforms Actions representation when predicting values of unseen policies in both real robotics scenarios-RGe RGB stacking and insert-gear (real)-as shown in Table I In the former, y VIT achieves regret@1 1 of 0.036 compared to Actions' 0.074, with a relative improvement of 51%. In the latter, CLPP improves over Actions by achieving regret@1 I 0.267 compared to Actions' 0.578 and drastically outperform Actions in terms of correlation by achieving +0.618 compared to Actions' -0.545. #2vec performs robustly on insert gear (real) despite policies' performances for this task vary greatly (see supplementary for per-task policies performances).
We also evaluate our approach in the simulated counterpart Insert Gear (Sim). In this environment, CCLIP and TAP achieve @egret@1 of 0.314 and 0.359 respectively, compared to Actions 0.427. We underline the dichotomy between geometrical and semantic features: T'P performs best in terms of NMAE and Correlation, while CLLIP outperforms in Regret@1 1. These results highlight how various compare depending on setting, type of task, and policy performance.
(ii) When evaluating across multiple settings, selecting ø leads to better results. We compare 2vec with different foundation models across 12 Metaworld settings and 15 Kitchen settings. Table reports the average results across all settings for Metaworld and Kitchen. In Metaworld, we notice that Actions performs on par with CCLP y and their respective variations ACLIP and VITT, in terms of correlation and regret@ I, while our approach consistently outperforms Actions in terms of NMAE. As these domains have less state variability, Actions represent policies robustly. We test CLIP/ALLIP
8

---

Published as a conference paper at ICLR 2024
Table 3: We extend v2ve to the fully-offline setting and test it on Metaworld assembly task (left, right, and top). We report results and confidence intervals. In this setting, performances of all policies are unknown.
Representation
NMAE 4
Correlation 1
Regret @
Assembly (left)
FQE
0.338 10.062
0.125 #0.218
0.424:10.260
#2vcc
8.306 10.155
0.360
0.215::.079
Assembly (right)
FQE
0.270+0.08
-0.029+0.330
0.504=0.071
#2vec
2.116 0.0
0.154=0.115
0.319
Assembly (top)
FQE
0.322::0012
=0.56
0.609+0.222
#2vec
0.492:1.0006
0.555=0.1u6 0.149
and VIT/AVIT on previously evaluated policies for each task through cross-validation to identify the best feature encoder for the task in terms of regret@ 1. We report &E and BEST as the average results over the best among CLIP/VIT and ACLIP/AVIT. BEST-CLIP achieves regret@) 0.194 and correlation 0.351, outperforming Actions representation. We highlight that the choice is critical, since firindom_ using Moreover, 2vee with the best @ drastically improves, achieving regret@ of 0.153 compared to Actions 0.232. We notice similar improvements when evaluating on Kitchen's 's 15 settings. Table 2 compares choosing the BEST e w.r.t to VIT and CLIP, and against Actions. In Kitchen, V outperforms &CLIP and Actions, while BESTT achieves the overall best results.
mnndomly-initializen ResNet50 as feature extractor extractor-underpeee underperforms.
(iii) r2vec enables fully-offline policy selection. By directly modelling the relationship between successor features and returns, avoid the linear reward assumption of the original successor features work. This preferable since rewards are generally not linearly related to state features. However, this restricts method to settings where some policies' performance is known. To evaluate performance in fully-offline setting, we linear model the task reward f = (Ø(s), Wass) given 2 from the original successor features work. Next we predict policies returns R, ( Wusk). We compare our approach FQE in Table and find that while our method's 's return predictions are inaccurate (as evidenced by the high NMAE), still performs well in ranking policies (higher Correlation and lower Regret@) 1).
the state's 's feature representation d(s), in Equation
CONCLUSION
We presented m2vec, a framework for offline policy representation via successor features. Our method treats the policy as black box, and creates a representation that captures statisties of how the policy changes the environment rather than its idiosyncrasies. The representations can be trained from offline data, and leverage the pretrained features of visual foundation models to represent individual states of the environment. In our experiments, we represented policies by relying on visual features from semantic (CLIP), geometric (TAP), and visual (VTT) foundation models. We showed that #2vec outperforms previously used Actions based representations and generalizes to fully-offline settings. Overall, our experiments showcase the effectiveness and versatility of π 2vec in representing policies and its potential for various applications in reinforcement learning.
Moving forward, we acknowledge that finding the optimal combination of these elements remains an ongoing challenge. Future work should explore diverse foundation models, offline learning algorithms for successor feature training, and dataset choices. fine-tuning the feature encoder ó along with e interesting but pose challenges, as each feature encoder would specialize to predict features for specific policy, resulting in policy representations that are independent and not comparable. We leave end-to-end fine-tuning as future work. Integrating #vvec into AOPS framework (Konyushova et al. 2021) for enhanced offline policy selection is another intriguing avenue. Additionally, extending #2vec to augment the Generalized Policy Improvement (Barreto et al. 2017) in offline settings presents exciting research opportunities.

---

Published as a conference paper at ICLR 2024
REFERENCES
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor features for transfer in reinforcement learning, Advances in neural information processing systems, 30, 2017.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. International Conference on Machine Learning, pp. 4494444455 458. PMIR, 2017.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
Kianté Brantley, Soroush Mehri, and Geoff Gordon. Successor feature sets: Generalizing successor representations across policies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11774-11781, 2021.
Jonathan Chang, Kaiwen Wang. Nathan Kallus, and Wen Sun. Learning bellman complete repre- sentations for offline policy evaluation. In International Conference on Machine Learning, pp. 2938-2971. PMLR, 2022.
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural computation, 5(4):613 -24,, 1993.
Carl Doersch, Ankush Gupta, Larisa Markeeva, Adrià Recasens, Lucas Smaira, Yusuf Aytar, João Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point in video. arXiv preprint arXiv:2211.03728 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. URL net/ id=Yi ddbnIcbddTYTyTT
Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas. and Serkan Cabi. Vision-language models as success detectors. arXiv preprint arXiv:2303.07280, 2023.
Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. pp. 1447-1456, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R Zhang, Yutian Chen, Aviral Kumar, et al. Benchmarks for deep off-policy evaluation. arXiv preprint 1rXiv:2103.1659 2021.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:7248-7259 2020.
Abhishck Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint 190. 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Alexander Irpan. Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine. Off Off-policy evaluation via off- policy classification. Advances in Neural Information Processing Systems, 32, 2019.
10

---

Published as a conference paper at ICLR 2024
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397, 2016.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 1806. 10293, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimizationn.. In Yoshua Bengio and Yann LeCun (eds.), ICLR (Poster), 2015. URL httt:://dbl:: uni-trierr--rrrr del db/
conE/iclt/icl:2000 htt.:///mcr.....mmmmmm.m. htt.ghaaaiiai1111
Ksenia Konyushova, Yutian Chen, Thomas Paine, Caglar Gulcehre, Cosmin Paduraru, Daniel J Mankowitz, Misha Denil, and Nando de Freitas. Active offline policy selection. Advances in Neural Information Processing Systems, 34:24631-24644 2021.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp. 5639-5650. PMLR, Jul 2020.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Interna- tional Conference on Machine Learning, pp. 3703-3712. PMLR, 2019.
Alex Lee, Coline Manon Devin, Yuxiang Zhou, Thomas Lampe, Konstantinos Bousmalis, Jost To- bias Springenberg, Arunkumar Byravan, Abbas Abdolmaleki, Nimrod Gileadi, David Khosid, et al. Beyond pick-and- and--lacee: Tackling robotic stacking of diverse shapes. In 5th Annual Conference on Robot Learning. 2021.
Lucas Lehnert and Michael Littman. Successor features combine elements of model-free and model- based reinforcement learning. The Journal of Machine Learning Research, 21(1):003-8800 8082, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arKiv:2005.0/66 2020.
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. pp. 661-670, 2010.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off-policy estimation. Advances Neural Information Processing Systems, 31, 2018.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 12601, 2022.
Xinkun Nie, Emma Brunskill, and Stefan Wager. Learning when-to-treat policies. arXiv preprint arXiv:1905.09751, 2019.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint XXXx:2007700055. 2020.
Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. PMLR, 2021.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.

---

Published as a conference paper at ICLR 2024
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez. Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. Transactions Machine Learning Research (TMLR), 2022.
Chris Reinke and Xavier Alameda-Pineda. Successor feature representations. arXiv preprint 10.7:1, 2021.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self- self-predictive representations. arXiv preprint arXiv:2007.05999, 2020.
Mohit Sharma, Claudio Fantacci, Yuxiang Zhou, Skanda Koppula, Nicolas Heess, Jon Scholz, and Yusuf Aytar. Lossless adaptation of pretrained vision models for robotic manipulation. 2023.
Richard Sutton, A Rupam Mahmood, and Martha White. An emphatic approach to the problem of off-policy cemporal-differeee learning. The Journal of Machine Learning Research, 17(1): 2603-2631, 2016.
Georgios Theocharous, Philip S Thomas, and Mohammad Ghavamzadeh. Personalized ad recom- mendation systems for life-time value optimization with guarantees. 2015.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-wort werldwwrld: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100. PMLR, 2020.
Michael R Zhang, Tom Le Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Ziyu Wang, and Mohammad Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. arXiv preprint 13X77 2021.
12

---

Published as a conference paper at ICLR 2024
7 APPENDIX
7.1 DOMAINS
The Metaworld Yu et al. (2020) and Kitchen Gupta et al. (2019) domains are widely known in the literature. They contain many tasks from multiple views, however, the variability among tasks is low. For example, the robotic arm in Metaworld is initialized within a narrow set of positions, while in Kitchen the object positions are fixed. The task in real robot RGB Stacking [ee et al. (2021): RRed al. (2022) to stack the red object on top of the blue object with green object as a distractor, where objects are of various geometric shapes. This task is difficult because the objects have unusual shapes and may be positioned at any point the workspace. We also consider a challenging gear insertion task in sim and real where the objective is to pick up gear of certain size (from an arbitrary position in the workspace) in the presence of other gears and insert it onto specific shaft on the gear assembly base (arbitrary position in real, fixed in sim). We describe data and policies for each domain below.
Insert Gear (Sim), We use 18 policies for Insert Gear (Sim) task in the simulated environment. We take intermediate and the last checkpoint for each policy. We collect trajectories with T 300 steps from a single and train all f/x, on those trajectories. The state 8 consists of two images, one from a left camera and one from a right camera, and proprioception sensing. All the policies in this domain have the following architecture. Image observations are encoded using a (shared) ResNet, and proprioception embedded using an MLP. Then, two embeddings are concatenated and further processed by an MLP, followed by an action head.
Insert Gear (Real). The observable state consists of three points of view: a camera on the left of the basket, a camera on the right of the basket, and an egocentric camera on the gripper. The state also includes proprioception information about the arm position. The setup corresponds to the medium gear insertion task described in the work of Du et al. (023). We conduct experiments on the Insert Gear (Real) task on a real robotic platform by evaluating 18 policies. We collect a dataset of trajectories with a hold-out policy trained human demonstrations. Next, we train our set of policies on this dataset and we evaluate 72vec. The state and he policy architecture are the same as in Insert Gear (Sim).
RGB stacking. We use 12 policies trained with behavior cloning on previously collected dataset of demonstrations for RGB stacking task with a real robotic arm. Each policy is trained with a variety of hyperparameters. The state consists of images from the basket cameras, one on the left and one on the right, and a first person camera mounted on the end-effector, and proprioception sensing. For training 7 22vec, we adopted an offline dataset of trajectories. We collected the trajectories by running a policy trained on human demonstrations. Trajectory length varies between 800 and 1000 steps. We built the evaluation dataset Dean by sampling 5, 000 trajectories and then sampling one state from each of them. Policy architecture follows the description in Lee et al. (2021).
Metaworld. For Metaworld, consider 4 tasks: assembly. button press, bin picking, and drawer open. We use 3 points of views (left, right, and top), as specified in Sharma et al. (2023): Nair et al. (2022). For each task-camera pair, we adopt 12 policies as proposed by [Sharma et al. (023) for the particular task and point of view. The policies vary by the hyperparameters used during training (learning rate, seed, and feature extractor among NFnet, VIT, and ResNet), Next, we train a successor feature network o- for each policy π on cumulative dataset of demonstrations for all tasks and views. At evaluation, we build Dean by uniformly sampling one state from each demonstration.
Franka-Kitchens For Kitchen, we consider 5 tasks: Knob-on, Left door open, light on, microwave open, and door open with points of views: left, right, and top, as provided by previous works (Sharma et al. 2223) Nair et al. 2022). For each task and point of view, use human demonstrations provided by Fu et al. 2020). We also adopt policies { proposed by /Sharma et al. (2023). Each policy solves each task using proprioception and image information from a single point of view, and the policies vary by the hyperparameters used during training (learning rate, seed, and feature extractor among NEnet, ViT, and ResNet). Additional details can be found in Sharma al. (2023)
Table 3 reports mean and standard deviation of the expected return values for the policies under consideration. We highlight that Metaworld and Insert gear have high standard-deviation (standard
13

---

Published as a conference paper at ICLR 2024
Table 4: We report the mean return value and its standard-deviation for the policies for each environ- ment. For Metaworld and Kitchen, we take the average mean and standard-devation across all tasks and points of views. f Number of policies per task-camera pair.
Environment
N. policies
Return values
Insert-gear (sim)
18
0.60=0.51
Insert-geat (real)
18
2. 2.29
RGB Stacking
12
18..41+8.21
Metaworld Kitchen
12f
215.88+161.99
12
-47.86120.mm
π T(St)
:
St
U/N
PROPRIO
B bins
Figure 4: We implement dh as a neural-network. First, we encode state se-consisting observations and proprioception-ant and policy actions (st)) into feature vectors. Next, concatenate the features and input the resulting vector to multi-layer perceptron. the outputs a vector of B x N dimensions, where B number of bins of the distribution and N is the dimension of the feature vector d(s:). We reshape the output into matrix, where each row i represents a histogram of probabilities of size B for the successor feature th..
deviation is 75% more of the mean return value), as we have extremely good and extremely bad policies. On the contrary, return values for RGB Stacking and Kitchen vary less, i.e., most of policies for these environments achieve similar performance.
7.2 ARCHITECTURE AND TRAINING DETAILS
The architecture of successor features network (s) for policy v is illustrated in Figure The network takes state-action pairs as input; it encodes actions and proprioception with multi-layer perceptron, and visual observations using a ResNet50. When the environment observation consists of multiple images (e.g.. multiple camera views of the same scene). embed each image separately and average the resulting vectors. We concatenate the state and action encodings and process the resulting feature vector with MLP. Finally, the network (s) outputs vector of dimension RN x where N is the dimension of the feature vector ((s) represented as a distribution over B bins"] (s) returns N histograms, where each histogram 8/s approximates the distribution of the discounted sum of feature é; over policy For each environment, we inspect the range of values assumed by o to find the min (lower) and max (upper) bound of the histogram. At inference time, we take the expected value of each histogram to compute the successor features vector.
We train ( a) using an FQE with a distributional objective (L et al. 2019 Bellemare et al. 2017) Training the successor features network only requires offline data (separate for each domain) and does not require any online interactions. We train the network for 50, 000 steps for Metaworld and Kitchen and 100, 000 steps for RGB Stacking, Insert Gear (Sim), and Insert Gear (Real). We adopt
We B bins all experiments.
14

---

Published as a conference paper at ICLR 2024
Table 5: We conduct additional experiments on Metaworld environment when using a dataset of trajectories for training 2vec. As expected, enhancing the dataset leads to better performances. We report BEST-CLIP and BEST-VIT the average results when adopting the best feature encoder between CLIP/VIT and ACLIP/AVIT in terms of regret@1,
Representation
Dataset
NMAE
Correlation
Regret @ I
Actions
0.424 20.058
0.347
40.152
0.232 +0.078
AVIT
traeectories
0.296 #0.024
0.399
+0.128
0.214
ACLIP
trajectories
0.278 #0.014
0,469 +0.096
0.189 +0.075
BEST-6
trajectories
0.26910.011
0.507 40. 105
0.187 1:0073
AVIT
best
0.322 +0.029
0.447=0.122
0.191+0.074
ACLIP
best
0.274 1:0026
0.537 10.166
0.177 1:0.175
BEST-
best
0.231 1:0016
0.615 0.086
0.135 1:0052
different training steps because Metaworld and Kitchen are more predictable than RGB Stacking and Insert gear and less demonstrations are provided. We use the Adam optimizer (Kingma & Ba 2015) with a learning rate of 3e and discount factor of y = 0.99. For evaluation, we adopt 3-fold vross-validation in all experiments.
7.3 METRICS
We adopt three common metrics Fu et al. (202T): NMAE, correlation, regret@
Normalized Mean Absolute Error (NMAE) is defined as the difference between the value and estimated value of policy:
NMAE |V V*
(6)
where V is the true value of the policy, and V is the estimated value of the policy.
Regret@1 @ is the difference between the value of the best policy in the entire set, and the value of the best predicted policy. is defined as:
Regret @ = max V
max V
iET:N
JE(I:N)
Rank Correlation (also Spearman's p) measures the correlation between the estimated rankings of the policies' value estimates and their true values. be written as:
Corr Cov(V VEN)
Correlation and regret@1 1 are the most relevant metrics for evaluating 72vec on Offline Policy Selection (OPS), where the focus is on ranking policies based on values and selecting the best policy,
Regret@1 is commonly adopted assessing performances for Offline Policy Selection, as it directly measures how far off the best-estimated policy is to the actual best policy.
Correlation is relevant for measuring how the method ranks policies by their expected return value. By relying on methods that achieve higher correlation and thus are consistent in estimating policy values, researchers and practitioners can prioritize more promising policies for online evaluation.
On the other hand, NMAE refers to the accuracy of the estimated return value. NMAE is especially significant when aiming at estimating the true value of a policy and is suited for Offline Policy Evaluation (OPE), where are interested to know the values of each policy. We assess #2vec's ' representation in both settings, showing that v2vec consistently outperforms the baseline in both metrics. We improve the discussion metrics in the Appendix of the manuscript.
7.4 ADDITIONAL EXPERIMENTS
(iv) Correlation between task difficulty and d. We notice that policy performance varies widely in Insert Gear (Sim) and Insert Gear (Real), as most of the policies fail to solve the task (see
15

---

Published as a conference paper at ICLR 2024
Table 6:
We evaluate vev and Actions baseline when comparing intermediate checkpoints for of policies for Metaworld assembly task and left point-of-view. We highlight that Actionsrepresentations are similar in such scenario, leading to poor generalization when evaluating on unseen and potentially different policies.
Representation
NMAE
Correlation
Regret
Actions
0.724 2:02668
-0.189
10.137
0.034
#0.014
ACLIP
0.570 10.173
0.190 20.361
0.029 #0.034
Table 7: We conduct an experiment an -ft-m--n setting. We represent a set of policies that were trained for Metaworld assembly task from right point-of-view with Actiony and &ACLIP, Next, we evaluate how those representations predict the return values policies are fed with images from different point-of-view (left camera in our experiment). y ACLIP outperforms Actionsin terms of regret@1 and NMAE, supporting our intuition that #2vcc is robust in an distrr--istiitutinn setting.
Representation
NMAE
Regret @
Actions
0.363 10.055
0.475 10.100
ACLIP
0.227 +0.078
0.300 10.106
Table 8: We intuitively expect that T2vec represents random policies close in feature space w.r.t. Actions representations, as random policies do not change the environment in any meaningful way. We conduct experiment with random policies for Metaworld assembly-left to provide quantitative evidence supporting our interpretation.
Representation
Average distance
Max distance
Actions
0.39
0.62
Random
0.11
0.17
ACLIP
0.03
0.22
supplementary for per-task policies performances). The gap is evident when compared to the average return value for RGB Stacking. where standard deviation is negligible. Our intuition that in hard-to-solve scenarios the actions are often imperfect and noisy. This interpretation would explain poor performance of the Actions baseline. The comparison of CCLIP and v VIT across environments suggests correlation between the choice of & and policies return values. &CLPP performs better than VIT in Insert Gear (Sim), Insert Gear (Real), and Metaworld, where we report the highest standard deviation among policies performances. UCLIP is robust when the task is hard and most of the policies fail to solve it. On the other hand, y VII, is the best option in Kitchen and RGB stacking, where the standard deviation of policies returns is low or negligible.
(v) Studying the performance of #2ec with different datasets. We investigate how modifications of the dataset for training #2vec improves performance in Metaworld. Intuitively, if the training set for 72vec closely resembles the set of reachable states for a policy π. solving Equation leads to a more close approximation of the real successor feature of π in (s, a). We empirically test this claim as follows. We collect 1, 000 trajectories for each task-view setting using pre-trained policy. Next, we train successor features network righ for each policy and feature encoder ø and represent each policy as x Table reports results on Metaworld when training #2vec with the aforementioned dataset (named trajectory in the Table). In this setting, CCLPP and VIT outperform both their counterpart trained on demonstrations and Actions representation, reaching respectively regret@) of 0.189 and 0.187. These results slightly improve if we assume to opt for the best dataset for each task and take the average. as reported under best dataset in Table 5 Overall, choosing the best feature encoder and best dataset for any given task leads to the best performing BESTT- with correlation 0.615 and @egret@1 0. 0.135-improving over Actions by 0.26 and 0.10 respectively.
(vi) 2ve generalizes better while Actions works with policies are very similar. We explore how Actions and T2vec compare in the special scenario where all policies are similar. We take 4 intermediate checkpoints the end of the training for each policy as a set of policies to represent.
16

---

Published as a conference paper at ICLR 2024
Table 9: We evaluate #2vec when trained on online data for Insert Medium (gear). Results for 22vec from offline data on this task are reported in Table
Representation
NMAE
Correlation
Regret@1 I
Actions
0.174
0.650
0.427
CLIP
0.158
0.627
0.288
Random
0.348
0.425
0.302
TAP
0.172
0.686
0.205
Our intuition is that intermediate checkpoints for a single policy are similar to each other in how they behave. Next, we represent each checkpoint with upCLPP and Actions. We compare cross-validating the results across all checkpoints w.r.t. training on checkpoints for 3 policies and testing on checkpoints for the holdout policy. Table&rrepore results of this comparison Metaworld's assembly-leff task. We notice that Actions representations fail to generalize to policies that greatly differ from the policies in the training set. Fitting the linear regressor with Actions achieves a negative correlation of -0.189 and regret 0.034. On the other hand, VCLPP is robust to unseen policies and outperforms Actions with positive correlation 0.190 and lower regret of 0.029.
(vii) 22ee performs in onttof-distribution scenarios. We conduct another investigation to explore 22vec performances in a oui-of-distribution setting. We hypothesize that #2ec represents policies in meaningful ways even when those policies are deployed in settings that differ from the training set, thanks to the generalisation power of foundation models. Table 7 compares ACLIP and Actions in evaluating policies trained for Metaworld's i assembly-right and tested in Metaworld's assembly- r2vec achieves reget@1 of 0.300 and NMAE of 0.227, outperforming Actions by 0. 0.175 and 0.136 respectively. We leave further exploration of 22ecc in out-of-distribution settings for further works.
(viii) =2vee represents random policies close in the representation space. Intuitively, we expect that random policies do not modify the environment in a meaningful way. Therefore, their representa- tions should be closer to each other compared to the similarity between the more meaningful trained policies. We investigate this claim follows. We provide a set of 6 trained policies and a set of random policies for Metaworld assembly-left. We compute the average and max distance among the random policies representations, normalized by the average intraset distance between trained policies representations. We compare our CCLPP and randomm with Actions. Table 8 reports the results that clearly support our intuition. Both v ACIIP and V Random represent random policies close each other, as the average distance of their representation is respectively 0.03 and 0.11 and the maximum distance 0.22 and 0. respectively. On the other hand, if we represent policies with Actions, the representations average and maximum distances are 0.39 and 0.62, meaning that random policies are represented far apart from each other.
(ix) Canonical state coverage. We sample canonical states uniformly from the dataset that was used for training offline RI policies. Even though there is some intuition that selecting canonical states represent the environment better can be beneficial, even simple sampling at random worked well in our experiments. We conduct further experiments to ablate the state coverage. We adopt demonstrations from Metaworld Assembly task and adopt the initial state of each trajectory for r2ee''s is and Actions representation. By adopting the initial state of trajectory, 72vec cannot rely on the state coverage. We report the results Table I0 We show that π 22ee is robust to state coverage, showing SOTA performances even when the canonical states coverage is limited to the first state of each demonstration.
(x) r2vec from ouline data. We ideally want to evaluate policies without deployment real robot, which is often time-consuming and can lead to faults and damages. Nonetheless, we explore 72vec capabilities for representing policies from online data. For each policy n. we collect a dataset of trajectories by deploying the policy on the agent. Next, we train d on the dataset of I 'ss trajectories and compute its #2vec's 's representation. Table on Insert Gear (sim) task. We show that 72vec's s performances improve with respect to the offline counterpart. This result is expected: a better dataset coverage leads to improved results, as we also showed in (v).
results when training 72vec on online data
17

---

Published as a conference paper at ICLR 2024
Table 10: We test r2ee robustness to canonical states coverage. We estimate #2vec's representations for Metaworld's assembly task (left, right, and top point-of-views) by using only the first state of the demonstrations for each task.
Representation
NMAE +
Correlation 1
Regret @
Assembly (left)
CLIP
0.381
0.359
0.287
ACLIP
0.260
0.592
0.087
Random
0.422
0.252
0.46
VIT
0.366
0.26
0.347
Actions
0.356
0.503
0.222
Assembly (right)
CLIP
0.363
0.023
0.365
ACLIP
0.242
0.582
0.096
Random
0.334
0.313
0,212
VIT
0.27
0.345
0.304
Actions
0.405
0.369
0.263
Assembly (top)
CLIP
0.463
0.270
0.330
ACLIP
0.305
0.594
0.078
Random
0.394
0.277
0.328
0.418
0.020
0,417
Actions
0.414
0.554
0.106
18


Reviews:
[{"review_id": "KFfoFAWsQH", "reviewer": "Reviewer_81Fm", "summary": "The ability to represent reinforcement learning policies in a vector space would allow for quickly evaluating policies' performance offline. Successor features is a learning paradigm that defines a policy's performance (i.e., expected reward) as being linear in the features of the policy. The authors propose to leverage ideas from successor features by predicting the performance of unknown policies with the performance of known policies based on joint learned successor feature embedding. This would allow for evaluating unknown policy's performance offline without needing online interactions.", "questions": "- If you had online policy trajectories from behavior policies, how would that affect the proposed idea?\n- How did you find the best policy to compare against for the regret metric? Was the best policy feature-dependent or feature-agnostic?\n- Why is the correlation metric related to how many evaluations on the reboot are required to find the best policy? I thought correlation was the relationship between the set of predicted values and ground-truth values. This relationship could be arbitrarily bad; no matter how many evaluations you do, the underlying feature may not provide a reasonable signal that relates to the ground truth.\n- Can I assume that correlation means relative performance (i.e.,  the ordering of values between the prediction and ground are the same), while NMAE is absolute performance (i.e., the predicted and ground values are exactly the same)?\n- What is significant of NMAE? The performance values of Table-1 and Table-2 imply that correlation indicates regret. This means that representations with low regret have a high correlation, but NMAE does not have a relationship to regret.\n- Why is action representation the only baseline applicable baseline? The underlying data has states and actions. Would it be unfair to condition a baseline on both state-action pairs?\n- The discussion from (i), (ii), and (iii) in the results section is confusing. In results (ii), the authors raise the point that NMAE is better, but in (iii), the authors raise the point that their approach is better in regret. What metric is the most important across these metrics presented?\n\n\nMissing cites:\n\n- Original Successor feature paper:  Improving generalization for temporal difference learning: The successor representation by Dayan, et al. 1993.\n- Successor Feature Representations by Reinke et al 2023\n- Successor Feature Sets: Generalizing Successor Representations Across Policies by Brantley et al. 2021\n- Successor Features Combine Elements of Model-Free and Model-based Reinforcement Learning by Lehnert et al 2020", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of storing an embedding space where you can look up policies could be beneficial for OPE and imitation learning.\n- The combination of exploring foundation model features and successor features is interesting.\n- The authors provide a very thorough empirical investigation of the performance of the proposed idea across several tasks. Furthermore, they investigated various representation learning ideas beyond changing the underlying foundational model, which provided insight into which representation is important for each task.\n- The evaluation metrics chosen provide insight between relative performance (i.e., correlation), absolute performance (i.e., NMAE), and performance against the best policy (i.e., regret). All three metrics captured three different and important aspects of the learning problem, providing a lot of insight.", "weaknesses": "- This paper needs more analysis regarding why certain features work better for certain settings. At the moment, if I implement this idea, I would have to enumerate all possible pairs.\n- The authors only compare to 1 baseline algorithm that only depends on the actions of the policies in the offline dataset. Meanwhile, the proposed method uses the state-action in the offline dataset to learn the successor feature components. I don't know if the performance increase of the baseline is due to the proposed idea or the additional information the proposed idea has access to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wh6hdgRzbB", "reviewer": "Reviewer_4YPE", "summary": "This paper proposed a new off-policy evaluation apporach. \nThe problem setting is that given a set of historical trajectories sampled from other policies and its corresonding performance, it predicts the performance of a unseen policy. \n\nThe main idea is using the average of successor features over a set of canonical states of each policy, to predict the policy performance.\nThe successor features are obtained via Fitted Q learning via a policy-agnositic state encoder.\n\nThe authors show that proposed approach is usually better than baseline across different domains and metrics.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper studied an important problem of off-policy evaluation.\n- As far as I k now, the idea of leveraging successor feature for the purpose of OPE is new.\n- The authors conducted extensive set of experiments.", "weaknesses": "- Potentially missing baseline: I am not closely follow the OPE literature, but could the authors explain why [1] is not suitable for this setting? \nWe should be able to get ranking results by that approach.\n- Missing key ablations: I think there are implicit key assumptions such as \n\t- 1) the historical policies' performance should mostly likely cover the unseen policy. If not, the unseen policy's performance is way better, I hardly expect this approach would work. \n\t- 2) the canonical states coverage. How does it affect the performance if state coverage of MDP is small? More specifically, if the historical dataset only cover 50% of\n\tstates that the unseen policy would visit in the MDP, how does it affect the performance?\n\n\tIt would be good to have a comprehensive understanding when this approach can be effective.\n- In section 3.4, \\Phi_\\pi represents an aggregated average effect of the behavior of π, I think this highly relies on the implicit assumptions mentioned above. It would be \ngood that authors can clarify.\n\n\n[1] Alexander Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine. Off-policy evaluation via off-policy classification. Advances in Neural Information Processing Systems, 32, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "75gtjRgkBn", "reviewer": "Reviewer_eE8e", "summary": "This paper proposes a method for representing reinforcement learning policies as comparable feature vectors. The idea is to use successor features to capture how a policy changes the states of the environment over time. In particular, the method leverages pretrained foundation models to encode individual states into feature vectors. After training the successor features on offline datasets of trajectories, the method aggregates the state-dependent successor features into a state-independent policy embedding that summarizes the policy's behavior. Finally, the embeddings are used to predict policy performance.", "questions": "1. Why do successor features plus foundation models work well for policy representation? More analysis on why this combination is effective compared to other representations would be useful.\n\n2. Can this method be extended to an online setting where policies interact with the environment? \n\n3. How does dataset composition impact the quality of the policy embeddings? Is there any further analysis on dataset requirements and relationships between dataset and representation quality?\n\n4. Have you considered any other methods to aggregate the state-dependent successor features? Averaging seems effective but overly simplistic; are there alternatives that may capture policies better?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel policy representation method: The combination of successor features and foundation models provides a new way to summarize and compare policies based on their effects on the environment. Representing policies by their induced state changes rather than their parameters or actions is an interesting idea.\n\n2. Strong empirical results: The paper presents extensive experiments across 5 domains, including real robots. The method outperforms the baseline policy representation method in predicting held-out policy performance.", "weaknesses": "1. Limited theoretical analysis: The paper shows empirically that the obtain successor representations are effective, but provides limited insight/intuition or analysis into why combining successor features and foundation models results in good policy embeddings.\n\n2. Restricted to offline setting: The method seems to require pre-collected offline datasets and cannot be applied in an online setting where policies interact with the environment. The offline assumption limits the applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F0YBppZ3ly", "reviewer": "Reviewer_Kqe9", "summary": "The paper presents an approach to build a compact, vectorial representation of policies, that can be used for off-policy evaluation. The idea is to leverage pre-trained image-based models to generate state-based features to be used for training successor features for policy evaluation. The policy features ($\\pi 2vec$) is obtained by averaging the successor features over a set of states sampled from the offline dataset.\n\nThe authors showed that these features are effective in the task of off-policy evaluation.", "questions": "See above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is clear and easy to follow. The approach is simple and intuitive, and, as far as I know, it is novel.", "weaknesses": "- I would appreciate more details about the relevance of this setting. What are the use cases you have in mind?\n- Could you clarify the metrics used to evaluate the approach? You should report (at least in appendix) the equations.\n- Why don't you use ranking metrics (Mean Average Precision, DCG, etc)? These seems quite relevant for evaluating methods for offline policy selection. As you mentioned, absolute error in terms of value prediction may not be always relevant.\n- You decided to focus on visual representation without clearly explaining why. I think it is important to evaluate the approach also on state-based observation. How would you select base state features in this setting?\n- Similarly, I would have expected an evaluation on standard offline benchmarks (e.g., D4RL or ExoRL). For example, if I'm not mistaken, ExORL provides also image-based observations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)