Self-Supervised Reinforcement Learning that Transfers using Random Features
Authors: Boyuan Chen
Chuning Zhu
Pulkit Agrawal
Kaiqing Zhang
Abhishek Gupta
Venue: NeurIPS 2023 poster (2023)
Keywords: deep reinforcement learning
self-supervised learning

Abstract: Model-free reinforcement learning algorithms have exhibited great potential in solving single-task sequential decision-making problems with high-dimensional observations and long horizons, but are known to be hard to generalize across tasks. Model-based RL, on the other hand, learns task-agnostic models of the world that naturally enables transfer across different reward functions, but struggles to scale to complex environments due to the compounding error. To get the best of both worlds, we propose a self-supervised reinforcement learning method that enables the transfer of behaviors across tasks with different rewards, while circumventing the challenges of model-based RL. In particular, we show self-supervised pre-training of model-free reinforcement learning with a number of random features as rewards allows implicit modeling of long-horizon environment dynamics. Then, planning techniques like model-predictive control using these implicit models enable fast adaptation to problems with new reward functions. Our method is self-supervised in that it can be trained on offline datasets without reward labels, but can then be quickly deployed on new tasks. We validate that our proposed method enables transfer across tasks on a variety of manipulation and locomotion domains in simulation, opening the door to generalist decision-making agents.

Self-Supervised Reinforcement Learning that Transfers using Random Features
Boyuan Chen"
Chuning Zhu" University of Washington Seattle, WA 98105
Massachusetts Institute of Technology Boston, MA 02139
boyuanc@mit.edd
edu
Pulkit Agrawal
Massachusetts Institute of Technology
Boston, MA 02139 pulkitagomit. edu
Kaiqing Zhang University of Maryland College Park, MD 20742 kaiqingeumd. edu
Abhishek Gupta
University of Washington Seattle, WA 98105 abhgupta@cs washington, edu
Abstract
Model-free reinforcement learning algorithms have exhibited great potential solving single-task sequential eecision-makingg problems with high-dimensional observations and long horizons, but are known to be hard to generalize across tasks. Model-based RL, on the other hand, Iearns task-agnostic models of the world that naturally enables transfer across different reward functions, but struggles to scale complex environments due to the compounding error. To get the best of both worlds, we propose a self-supervised reinforcement learning method that enables the transfer of behaviors across tasks with different rewards, while circumventing the challenges model-based RL. In particular, we show self-supervised pre- training of model-free reinforcement learning with number of random features rewards allows implicit modeling of long-horizon environment dynamics. Then, planning techniques like model-predictive control using these implicit models enable fast adaptation to problems with new reward functions. Our method is supervupe in that can be trained on offline datasets without reward labels, but can then be quickly deployed on new tasks. We validate that our proposed method enables transfer across tasks on variety of manipulation and locomotion domains in simulation, opening the door to generalist docision-making agents.
Introduction
As in most machine learning problems, the ultimate goal of building deployable sequential decision- making agents via reinforcement learning (RL) its broad generalization across tasks in the real world. While reinforcement learning algorithms have been shown successfully synthesize complex behavior in single-task sequential decision decision-making problems [31 36 45). their performance as generalist agents across tasks has been less convincing 99. In this work, focus the problem of learning generalist agents that are able to transfer across problems where the environment dynamics shared, but the reward function is changing. This problem setting is reflective of scenarios
"Equal contribution. I Equul advising.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

that may be encountered in real-world settings such as robotics. For instance, in tabletop robotic manipulation, different tasks like pulling an object. pushing an object, picking it up, and pushing to different locations, all share the same transition dynamics, but involve different reward functions. We hence ask the question- Can we reuse information across tasks in a way that scales to high dimensional, longer horizon problems?
offine exploration trajeciories without reward
online quick adaptation to task
RaMP
π (-lss)
Figure I:
RaMP sele--upervised reinforcement learning algorithm that pre-training on exploration trajectories
without (or with unknown) reward. It then quickly adapts to new tasks through interaction with the environment.
Typical policy search algorithms can achieve good performance on a single task. However, the policy is optimized for particular reward and may be highly suboptimal in new scenarios. Other model-free RL algorithms like actor-critic methods [23 15 [3] or Q-learning [51 31 may exacerbate this issue, with learned
natural possibility to tackle this problem is direct policy search 54
44
Q-functions entangling dynamics, rewards, and policies. For new scenarios, an ideal algorithm should be able to disentangle and retain the elements of shared dynamics, allowing for easy substitution of rewards without having to repeat significant computations. While multi-task policy search techniques like goal-conditioned RL address this issue to some extent, they restricted to particular tasks like goal-reaching and usually require prior knowledge about the tasks one may encounter in testing.
Model-based RL arises natural fit for disentangling dynamics and rewards
33
10
19
20
These algorithms directly learn a model of transition dynamies and leverage the learned model to plan control actions B5 17 10, g In particular, the models can be used to re-plan behaviors for new rewards. However, these learned dynamics models are brittle and suffer from compounding error [25 3 due to their autoregressive nature. States predicted by the model are iteratively used as inputs for subsequent predictions, leading to compounding approximation errors during planning (25). This makes it challenging to apply them to problems with long horizons and high dimensions.
In this work, we thus ask Can we build RL algorithms that disentangle dynamics, rewards, and policies for transfer across problems, while retaining the ability to solve problems with high dimensional observations and long horizons? To this end, we propose a self-supervised RL algorithm that can leverage collected data to implicitly model transition dynamics without ever having to generate future states, and then use this to quickly transfer to a variety of different new tasks with varying reward functions that may be encountered at test time.
Our method relies the following key insight: to prevent tying specific reward functions and policies to the RI. algorithm, one can instead model the long-term evolution of randomly chosen basis functions of all possible reward functions (6. Such basis functions can be generated using random features [19 The accumulation of such basis functions over time, which we refer to as a Q-basis, task-agnostic [48], and implicitly models the transition dynamics of the problem. We show that with large enough number of random features and model-free RL, the value function for any reward function that may be encountered in testing could be estimated, via a simple linear combination of these Q-basis functions. This is important since it allows us to obtain the benefits of transfer, without explicitly learning a dynamics model that may suffer from compounding error. Moreover, with randomly chosen basis functions that are agnostic of downstream tasks, this is not restricted to just a certain distribution of tasks, as most multi-task or goal-conditioned RI. 49 2 Finally, we note that the random features (as well as the corresponding Q-basis functions) can be generated using datasets that are not necessarily labeled with reward signals. Indeed, these random-feature based pscudo rewards can be viewed as self-supervised signals in training. making the approaches based on them naturally compatible with large aankstagnsticc offline datasets.
Based on the aforementioned insights, we propose new algorithm Random Features for Model-Free Planning (RaMP) that allows us to leverage unlabelled offline datasets to learn reward- awaar-agnosticc Q- bases. These can be used to estimate the test-time value functions for new reward functions using linear regression, enabling quick adaptation to new tasks under the same shared transition dynamics. We show the efficacy of this method on number of tasks for robotic manipulation and locomotion in simulation, and highlight how RaMP provides a more general paradigm than typical generalizations of model-basee or model-free RL.

---

1.1 Related Work
Model-based RL is naturally suited for transfer across tasks with shared dynamics, by explicitly learning a model of the transition dynamics, which is disentangled with the reward functions for each task [17 35 10, 16, 8 47 27]. These models are typically learned via supervised learning on one-step transitions and then used to extract control actions via planning [28] 42 or trajectory optimization 46 59 38). The key challenge in scaling lies in the fact that they sequentially feed model predictions back into the model for sampling 500 25 This can often lead to compounding errors [25 B 551, which grow with the horizon length unfavorably. In contrast, our work does not require autoregressive sampling, making easier scale to longer horizons and higher dimensions.
On the other hand, model-free RL avoids the challenge of compounding error by directly modeling either policies or Q-values without autoregressive generation [54 43 44
R1 [5]. However, these approaches entangle rewards, dynamics, and policies, making them hard to transfer. While attempts have been made build model-free methods that generalize across rewards, such as goal-conditioned value functions [ 37 14] or multi-task policies (22 18] they only apply to restricted classes of reward functions and particular training distributions. Our work aims to obtain the best of both worlds (model- -ased and model-free RL), learning some representation of dynamics independent of rewards and policies, while using a model-free algorithm for learning.
30
Our notion of long-term dynamics is connected to the notion of state-action occupancy measure 588. often used for off-policy evaluation and importance sampling methods in RL. These methods often try to directly estimate either densities or density ratios (20 34 [58) Our work simply learns the long-term accumulation of random features, without requiring any notion of normalized densities. Perhaps most closely related work to ours is the framework of successor features, that considers transfer from fixed set of source tasks to new target tasks (5 26 57 [I]. Like our work, the successor features framework leverages the linearity of rewards to disentangle dynamics from rewards using model-free RL. However, transfer using successor features is dependent on choosing (or learning) the right featurization and forces an implicit dependence on the policy. Our work leverages random features and multi-stey Q-functions to alleviate the transfer performance successor features.
34
Preliminaries
Model. We consider the standard Markov decision process (MDP) as characterized by tuple (S,A, T, R. R.Y,H)), with state space S, action space A. transition dynamics T SSxAA> A(S), reward function R : S x A + A([-Rmax: Rmax)), ), discount factor 7 [0, 1). and initial state distribution E (S)). The goal is to learn a policy π S - A (A), such that it maximizes the expected discounted accumulated rewards, i.c., solves max, Eg - >^'rr,] with 77 ;= r(s,, 0n) ~ Ran.as = Pr(-I Sh, ax). Hereafter, we will refer to an MDP and task interchangeably. Note that in our problem settings, different MDPs will always share transition dynamics T, but will have varying reward functions R.
Q-function Estimation. Given MDP M, one can define the state-action Q-value function under any policy क as (=( == E encva(-
E 81 = 8, 01 a which denotes
Sh+r~T(- (s..o))
the expected accumulated reward under policy n, when starting from state-action pair (s, a). By definition, this Q-function is inherently tied to the particular reward function R and the policy n making it challenging transfer for new reward policy. Similarly, one can also define the multi-
step (T-step) Q-function Qx (8, ãi, ".,... ,ū+) = Barth -xx(- (844))
E 7 b-1rn
8,,1
Snt1a (8h0n)
ã1, 02
07
One can estimate the stsppp Qx by Monte-Carlo sampling of the trajectories under æ,, i.e., by solving
M
min QEo
N Σ
100. ... a))
(2.1)
M
where Q is some function class for Q-value estimation, which in practice is some parametric function class, e.g., neural networks; rh m.j
~ R mis mis and (s) mj m.j) come from NN trajectories that agh
3

---

are generated by N action sequences {[a], " , and M trajectories following policy π after each action sequence. A large body of work considers finding this Q-function using dynamic programming, but for the sake of simplicity, this work will only consider Monte-Carlo estimation.
In practice, the infinite-horizon estimator in (2.1) can be hard obtain. We hence use finite-horizon approximation of Q2 (of length H), denoted by Q in learning. We will treat this finite-horizon formula in the main paper for practical simplicity. We also provide a method to deal with the infinite-horizon case directly by bootstrapping the value function estimate. We defer the details to Appendix B.1 Note that if one chooses H = then the T-step Q-function defined above becomes Q (s, 01, .2, ,uH)
mn) L-m > $1 5, 5 ",,-.. , ,H aæ Note that in this case, the Q-function is irrelevant of the policy я, denoted by Q", and is just the expected accumulated reward executing the action sequence (ā) ",,"".... ... a starting from the state This Q-function can be used to score how "good" sequence of actions will be, which in turn can be used for planning.
Problem setup. We illustrate our problem setup in Figure Consider a transfer learning scenario, where we assume access to an offline dataset consisting of several episodes 'D a (8+1)) + Meem).MMMM Here H is the length of the trajectories, which is large enough, e.g., of order O(1((1 (1-111))))))))))))))((()))) -)) to approximate the anfinite-horizon setting: M is the total number of trajectories. This dataset assumes that all transitions are collected under the same transition dynamics T. but otherwise does not require the labels, i.c., rewards, and may even come from different behavior policies. This is reminiscent of the offline RL. problem statement, but offline RI. datasets are typically labeled with rewards and are task- In contrast, in this work, the pre-collected dataset is used to quickly learn policies for any downstream reward, rather than for one specific reward function. The goal is to make the best use of the dataset D. and generalize the learned experience to improve the performance on a new task M, with the same transition dynamics T but an arbitrary reward function R. Note that unlike some related work (5 4 we make no assumption on the reward functions of the MDPs that generate 'D. The goal of the learning problem is to pre-train on the offine dataset such that we can enable fast (even zero-shot) adaptation arbitrary new reward functions encountered at test time.
3 RaMP: Learning Implicit Models for Self-Supervised Model-Free RL
Transfer with
We now introduce our algorithm, Random Features for Model-Free Planning (RaMP), to solve the problem described in Section -ecarinn-learninn learning a model of long-term dynamics that enables transfer to tasks labeled with arbitrary new rewards, while making use of the advantages of model-free RL. Clearly, the problem statement we wish solve requires us to (1) be able to solve tasks with long horizons and high- -ighh-imensional observations and (2) alleviate policy and reward dependence to be able to transfer computation across tasks. The success of model-free RI. in circumventing compounding error from autoregressive generation raises the natural question: Is there model-free approach that can mitigate the challenges of compounding error and can transfer across tasks painlessly? We answer this in the affirmative in the following section.
3.1 Key Idea: Implicit Model Learning with Random Feature Cumulants and Model-Free RL
The key insight we advocate is that avoid task dependence if we directly model the long-term accumulation of many random functions of states and actions (treating them as the rewards). instead of modeling the long-term accumulation of one specific reward as in typical Q-value estimation. Since these random functions of the state and actions are task-agnostic and uninformed of any specific reward function, they simply capture information about the transition dynamics of the environment. However, they do so without actually requiring autoregressive generative modeling, as commonplace in model-based RL. Doing so can effectively disentangle transition dynamics from reward, and potentially allow for transfer across tasks, while still being able to make use of model-free RL methods. Each long-term accumulation of random features is referred to as an element of a "random" Q-basis, and can be learned with simple modifications to typical model-free RI. algorithms. The key is to replace the Q-estimation of a single task-specffic reward function with estimating the Q- f-functions of a set of task-agnostic random functions of the state and actions a random Q-bssss.

---

At training time, the offline dataset D can be used to learn set of "random" Q-basis functions for different random functions of the state and action. This effectivvly forms an "implicit model", as carries information about how the dynamics propagate, without being tied to any particular reward function. At test time, given a new reward function, we can recombine Q-basis functions to effectively approximate the true reward-specinf Q- Q-function under any policy. This inferred Q-function can then be used for planning for the new task. As we will show Section 3.4, this recombination can actually be done by a linear combination of Q-basis functions for sufficiently rich class of random features, reducing the problem of test-time adaptation for new rewards to a simple linear regression problem. We detail the two phases of our approach in the following subsections.
Offline Training
Online Quick Adaptation to New Objective(s)
Fity y acoumulated features over trajectoyy
18 dil
1 (g)
(s dgg (g))
exploration Injectories oof unknown reword
W Q-basis
Small Amouni of Interaction
Linear Regression
Random Features
D
50
w that thy w
approximates
(w. ww..b(ss.e
at 02 ww..k
(g))
a)),,a)) + (+2,, az)... ( 4y) Q(s, dy diy an))
for
respect patential function
Model Predictive Control
Infer Qeest (8,0,, ag)
Figure 2: RaMP: Depiction of our proposed method for transferring behavior across tasks by leveraging model- free learning of random features. At training time, Q- basis functions are trained on accumulated random features. At test time, adaptation is performed by solving linear regression and recombining hasis functions, followed by online planning with MPC.
3.2 Offline Training: Self-Supervised Learning Random Q-functions
Without any prior knowledge about the downstream test-time rewards, the best that agent can do is to model the evolution of the state (i.c., model system dynamics). The key insight we make is that the evolution of state can be implicitly represented by simply modeling the long-term accumulation of random features of state to obtain a set of Q-basis functions. Such a generation of random features is fully self-supervised. These functions can then be combined to infer the task-specific Q-function.
Given the lack of assumptions about the downstream task, the random features being modeled must be expressive and universal in their coverage. As suggested in [39H41), random features can be powerful that most nonlinear functions can represented as linear combinations of random features effectively. As will show in Section 3.4 modeling the long-term accumulation of random features allows us express the value function for any reward as a linear combination of these accumulations of random features. In particular, in work, assume that the random features represented as the output of K neural networks ø(, :; 0x) S x A + with weights 0g E IR and k E [K], where 0g are randomly sampled i.i.d. from some distribution p. Sampling K such weights 0% with l E [K] yields a vector of sealar functions [,(:;:; :; 0;x) [ E TR& for any (s,a), which can be used random features. To model the long-term accumulation of each of these random features, we note that they can be treated as reward functions (as can any function of state and action [48]) in model-free RL, and standard model-free policy evaluation to learn Q-functions can be reused to learn a set of K Q-basis functions, with each of them corresponding to the Q-value of a random feature.
We note that done naively, this definition of Q-basis function is tied to particular policy that generates the trajectory. However, to transfer behavior one needs to predict the accumulated random features under new sequences of actions, evaluating a policy that is not the same as the training policy π. To allow the modeling of accumulated features that is independent of particular policies, we propose learn multi-step Q-basis functions for each of the random features (as discussed in Section which is explicitly dependent on an input sequence of actions (ā,, ................................ ,ā)) This can be used to search for optimal actions in new tasks.
policy search for the new task is likely to involve
To actually learn these Q-basis functions (one for each random feature), we use Monte-Care methods for simplicity. We generate new dataset Do from D, with Do {((s]) aHa), Σne(m ~ ah (e)))me(M))eEE) Here we use
5

---

EnE(H - 6(s% ax; 0s) as the accumulated random features for action sequences fai,'.. (H) taken from state 81- We then K function approximators, representing each of the K Q-basis functions, e.g., neural networks vk:))2)) S x AA + for k [K], to fit the accumulated random features. Specifically, we minimize the following loss
min XI EmE[M),AE[K (+(+T)
(1"""" "THH uk) Eneim "" aa:::ag;;00 0s))
This objective is Monte Carlo estimate, but we can also leverage dynamic programming to do so in an infinite-horizz way, as we show in Appendix B.1
3.3 Online Planning: Inferring Q-functions with Linear Regression and Planning
Our second phase concerns online planning to transfer to new tasks with arbitrary rewards using our learned Q-basis functions. For a sufficiently expressive of random non-linear features, any non-linear reward function can be expressed as a linear combination of these features. As we outline below, this combined with linearity of expectation, allows us to express the task-speeific Q-function for any particular reward as a linear combination of learned Q-bases of random functions. Therefore, we can obtain the test-time Q-function by solving simple linear regression problem from random features to instantaneous rewards, and use the same inferred coefficients recombine Q-bases to obtain task-specific Q-functions. This inferred Q-function can then be used to plan an optimal sequence of actions. We describe each of these steps below.
3.3.1 Reward Function Fitting with Randomized Features
We first learn how to express the reward function for the new task as linear combination of the random features. This can be done by solving linear regression problem to find the coefficient vector w = [w], wx] that approximates the new task's s reward function as a linear combination of the (non-linear) random features. Specifically, we minimize the following loss
w argminn
1 MH
(r(O)) 0% ME(M)
Σ
Σ ME[K]]
wro(sK (akkss"aaaa" ah; 0n)) + A||w||
(3.1)
where À 2 0 is the regularization coefficient, and r(s), ;a,) ~ Rum .am Due to the use of random features, Eq. (3.1) is ridge regression problem, and can be solved efficiently.
Given these weights, it is easy to estimate an approximate multi-step Q-function for the true re- ward the task by linearly combining the Q-basis functions learned in the offline train- ing phase (f)
according to the same coefficient vector w This follows from the additive nature of reward and linearity of expectation. In particular, if the reward function r(s,a, a) = Eke[K] wgo(s, a; (g) holds approximately. which will be the case for large enough K and rich enough d, then the approximate Q-function for the true test-time reward under the sequence (ar,".,, ,an) satisfies Q#1 (s1,a1.1)
(ss,s~ [>e(H] " 2
EkE[K] wirb ai.s; vē), where [wā kejk] is the solution to the regression problem (Eq. (3.1)
and {vf ] is the solution to the Q-basis fitting problem described in Section 3.2
3.3.2 Planning with Model Predictive Control
To obtain the optimal sequence of actions, true reward Q (81, (1.H1) ) for online planning at each time / in the new task: at state 8t, we conduct standard preei-predictive control techniques with random shooting 35 52 i.e., randomly generating N sequences of actions {a],- .... .a } N and find the action sequence with the maximum Q-value such that n E argmax,e(N) ΣλE[K] (hs,, 4f1+
the inferred approximate Q-function for the
We note that this finite-horizon planning is an approximation of the actual infinite-horizon planning, which may be enhanced by an additional reward-to-go term in the objective. We provide such an enhancement with details in Appendix B.T although we found minimal effects on empirical performance. We then execute a refer readers to Appendix[i For detailed connection of our proposed method to existing work and Appendix A for pscudocode.
"i from the sequence ni, observe the new state 8 1, and replan. We

---

3,4 Theoretical Justifications
We now provide some theoretical justifications for the methodology we adopt. To avoid unnecessary nomenclature of measures and norms in infinite dimensions, we in this section consider the case that S and A are discrete (but can be enormously large). Due to space limitation, we present an abridged version of the results below, and defer the detailed versions and proofs in Appendix D the following result on the expressiveness of random cumulants.
We first state
Theorem 3.1 (Q-function approximation; Informal). Under standard coverage and sampling assump- tions of offline dataset D and standard assumptions the boundedness and continuity of random
features (8, a; 0), follows that with horizon length H O ( 1-7
[mxx/s) and - ((--)))))
episodes in dataset D, and with K = =i(11 - 7) - 2) random features, we have that for any given reward function R. and any policy
1i"" (ww) 1x S 50((e) ( inf E(f) IEH where Q41 (s, w*)
with E E[E 7
high
probability,
is defined as
ExE[K] wK an 0%) [1
s, 01 = a] for each (s, and can be esti-
mated from the offline dataset D; jnf E(f is the infimum expected risk over the function class H. induced by ø.
Theorem i. is an informal statement of the results in Appendix D.2 which specifies the number of random features, the horizon length per episode, and the number of episodes, in order to approximate Qx accurately by using the data in D, under any given reward function R and policy æ. Note that the number of random features is not excessive and is polynomial in problem parameters. We also note that the results be improved under stronger assumptions of the sampling distributions p and kernel function classes [4]
Next, we justify the use of multi-step Q-functions in planning in deterministic transition environ- ment, which contains all the environments our empirical results we will evaluate later. Recall that for any given reward R, let Q, denote the Q-function under policy Note that with slight abuse of notation, Qr can also be the multi-step Q-function (see definition in Section 2 and the meaning should be clear from the input, i.e., whether it is (s, a) (x, a1. ,0H)
Theorem 3.2. Let II be some subclass of Markov stationary policies, i.c., for any II, T S (AA) Suppose the transition dynamics T is deterministic. For any given reward R. denote the H-step policy obtained from H-step policy improvement over TI as THH S -> A" defined as TH(() (s) E argmax A max (= 01, ,uH,, for all E S. Let Vaig denote the value function under the policy Fh (see formal definition in Appendix D.I. Then, we have that for all s E S
V (+) 2 max max Q=(s, #1: well
.nnn 2 max max (s. xlll
The proof of Theorem 3.2 can be found in Appendix D.2 The result can be viewed as a generalization of the generalized policy iteration result in [5] to suep action-sequence policies. Taking II to be the set of policies that generate the data, the result shows that the value function the greedy multi-action policy improves over all the possible H-step multi-action policies, with the policy after step H to be any policy in II. Moreover, the value function by TH also improves overall one-step policies if the policy after the first step onwards follows any policy II. This is due the fact that II (coming from data) might be a much more restricted policy class than any action sequence @1.H.
Experimental Evaluation
In this section, we aim to answer the following research questions: (1) Does RaMP allow for effective transfer of behaviors across tasks with varying rewards? (2) Does RaMP scale to domains with high-dimensional observation and action spaces and long horizons? (3) Which design decisions in RaMP enable better transfer?
4.1 Experiment Setup
Across several domains, we evaluate the ability of RaMP to leverage the knowledge shared dynamics from an offline dataset to quickly solve new tasks with arbitrary rewards.
7

---

Figure 3: We evaluate our method on manipulation, locomotion, and high dimensional action space environ- ments. The green arrow in each environment indicates the online objective for policy transfer while the red arrows are offline objectives used to label rewards for the privileged datasct. Reach Wall
Doar Open
Faucet Open
Buttonn Press
0.8
255
20
9.6-
Edcods
1.5
1.0
0.2
0.5
0.5
Hopper Stand
1s4
Hopper Jump
le4
Hopper Backward
let
D'Clow
lel
2.5
-5.0
Envrorment Steps
1st
Enviromment Steps
led
Environment Steps
let
Environment Steps let
RaMP (Durs)
MEPO
PETS
SF
COL
Figure 4:
Reward transfer results on Metaworld, Hopper and D' Claw environments. RaMP adapts to novel rewards more rapidly than MBPO, Successor Features, and CQL baselines.
Offline Dataset Construction: For each domain, we have an offline dataset collected by a behavior policy described in Appendix C.2 Typically this behavior policy is mixture of noisy policies accomplishing different objectives in each domain. Although RaMP and other model-based methods do not require knowledge of any reward from the offline dataset, other baseline comparisons will require privileged information. Baseline comparison methods like model-free RI. and successor features require the provision of objectives on state actions from the offline dataset. We call such objectives "offline objectives" and a dataset annotated with these offline objectives and rewards privileged dataset.
of training objectives,
well rewards labeled for these
Test-time adaptation: At test-time, we select a novel reward function for online adaptation, referred to 'online objective below. The online objective may correspond to rewards conditioned on different goals or even arbitrary rewards, depending on the domain. Importantly, the online objective need not be drawn from the same distribution as the privileged offline objectives.
Given this problem setup, we compare RaMP with a variety of baselines. (1) MBPO 19 model- based RI. method that learns standard one-step dynamics model and uses actor-critic methods to plan in the model. We pre-train the dynamics model for MBPO on the offline dataset. (2) PETS model-based RL method that learns ensemble of one-step dynamics models and performs MPC. We pre- pre-train the dynamics model on the offline dataset and use the cross-entropy method (CEM) to plan. (3) Successor feature (SF) (5) is framework for transfer learning in RL as described in Sec. [.] SF typically assumes access to a set of policies towards different goals along with learned featurization, we provide it with the privileged dataset to learn set of training policies. We also learn successor features with the privileged dataset. (4) CQL 24]: As an oracle comparison, we compare with coal-conditioned variant of an offline RL algorithm (CQL). CQL is model-free offline RI algorithm that learns policies from offline data. While model-free offline RI. naturally provide CQL with information about the goal
struggles to adapt arbitrarily changing rewards, at both training and testing time. CQL is trained on the distribution of training goals on the offline dataset, and finetuned on the new goals test time. The CQL comparison is assuming access to more information than RaMP. Each method is benchmarked on each domain with seeds.
4.2 Transfer to Unseen Rewards
We first evaluate the ability of RaMP to learn from an offline dataset and quickly adapt to unseen test rewards in 4 robotic manipulation environments from Meta- World [56]. We consider skills like

---

reaching target across the wall, opening door, turning on a faucet, and pressing button while avoiding obstacles, which are challenging for typical model-based RL algorithms (Fig. [}. Each domain features 50 different possible goal configurations, each associated with a different reward but the same dynamics. The privileged offline objectives consist of 25 goal configurations. The test-time reward functions are drawn from the remaining 25 "itt-o--disribution" reward functions. We refer the reader to Appendix
[CI for details.
As shown in Fig 4 our method adapts to test reward most quickly across all four meta-world domains. MBPO slowly catches up with our performance with more samples, since it still needs to learn the Q function from scratch even with the dynamics branch trained. PETS adapts faster than MBPO but performs worse than our method as it suffers from compounding error. In multiple environments, successor features barely transfer to the online objective as it entangles policies that are not close to those needed for the online objective. Goal-cconitionee CQL performs poorly all tasks as it has a hard time generalizing to out-of-distributi goals. In comparison, RaMP is able to deal with arbitrary sets of test time rewards, since does not depend on the reward distribution at training time.
4.3 Scaling to Tasks with Longer Horizons and High Dimensions
We further evaluate the ability of our method to scale to tasks with longer horizons. We consider locomotion domains such as the Hopper environment from OpenAl Gym (7]. We chose the offline objectives to be a mixture of skills such as standing, sprinting, jumping, or running backward with goals defined velocities or heights for standing, sprinting, jumping or running. As shown in Fig. 4 our method maintains the highest performance when adapting novel online objectives, as it avoids compounding errors by directly modeling accumulated random features. MBPO adapts at a slower rate since higher dimensional observation and longer horizons increase the compounding error of model-based methods. We note that SF is performing reasonably well, likely because the method also reduces the compounding error compared to MBPO, and it has privileged information.
target velocities or heights. The online objectives consist of unseen goal
To understand whether RaMP can scale to higher dimensional state-action spaces, we consider a dexterous manipulation domain (referred to as the D D'Claw domain in 3). This domain has a 9 DoF action space controlling each of the joints the hand as well as a 16-dimensional state space including object position. The offline dataset is collected moving the object to different orientations, and the test-time rewards are tasked with moving the object to new orientations (as described in Appendix C.Ipp Fig 4 shows that both Q-estimation and planning with model-predictio control remain effective when action space is large. In Appendix C.3 we test our method on environments with even higher dimensional observations such as images.
4.4 Ablation of Design Choices
To understand what design decisions in RaMP enable better transfer and scaling, conduct ablation studies on various domains, including an analytical 2D point goal-reaching environment and its variants (described in Appendix C.I), well as the classic double pendulum environment and meta-world reaching. We report an extensive set of ablations in Appendix C.5
Reduction of compounding error with multi-step Q functions
We hypothesize that our method does not suffer from compounding errors in the same way that feedforward dynamics models do.
Hopper
Pendulum
Ours MBRL
25.7 + 5.4
65.1 + 0.4 395.4 + 5.8
In Table I we compare the approximation error of trun- cated Q values computed with (1) multi-step Q functions obtained as linear combination of random cumulants (Ours), and (2) rollouts of dynamics model (MBRL). We train the methods on the offline data and evaluate on the data from a novel task test time. As shown in Table our method outperforms feedforward dynamics models.
50.2 + 9.7
Table 1:
Policy evaluation error. Feed- forward dynamics model suffers from com- pounding error that is particularly noticeable in domains with high action dimensions or chaotic dynamics. Our method achieves low approximation error in both domains.
5 Discussion
In this work, we introduce RaMP, method that leverages diverse unlabeled offline data to learn models of long horizon dynamies behavior, while being able to naturally transfer across tasks with different reward functions. To this end, we propose to learn the long-term evolution of random features

---

under different action sequences, which can be generated in self- supervise fashion without reward labels. This way, we are able to disentangle dynamics, rewards, and policies, while without explicitly learning the dynamic model. We validate our approach across a number of simulated robotics and control domains, with superior transfer ability than baseline comparisons. There are also limitations of the current approach we would like to address in the future work: the current Monte-Carlo- value function estimation may suffer from high variance, and we hope to incorporate the actor-critic algorithms [23] in RaMP to mitigate it; current approach is only evaluated simulated environments, and we hope to further test on real robotic platforms.
Acknowledgments
We would like to thank Russ Tedrake, Max Simchowitz, Anurag Ajay, Marius Memmel, Terry Suh, Xavier Puig. Sergey Levine, Seohong Park and many other members of the Improbable Al lab at MIT and the WEIRD Lab at University Washington for valuable feedback and discussions.
References
[1] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O, P. Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pages 5048-5058, 2017.
[2] M. Andrychowicz. F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin,
P. Abbeel, and W. Zaremba. Hindsight experience replay. CoRR, Abs/1707.00145, 2017.
[3] K. Asadi, D. Mísra, and M. L. Littman. Lipschitz continuity in model-based reinforcement learning. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässsan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 264-273, PMLR, 2018.
[4] A. Barreto, D. Borsa, J. Quan, T. Schaul, D. Silver, M. Hessel, D. Mankowitz, A. Zidek, and R. Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pages 501-510. PMLR, 2018.
[5] A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. P. van Hasselt, and D. Silver. Successor features for transfer in reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
[6] E. Bingham and H. Mannila. Random projection in dimensionality reduction: applications to image and text data. In Knowledge Discovery and Data Mining, 2001.
[7] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba. Openai gym. CoRR, abs/1606.015444 2016.
[8] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. CoRR, abs/1805.12111 2018.
[9] K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman. Quantifying generalization in reinforcement learning. CoRR, abs/1812.02341, 2018.
[10] M. P. Deisenroth and C. E. Rasmussen. PILCO: A model-based and data-efficient approach to policy search. L. Getoor and T. Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning, ICML 20/ Bellevue, Washington, USA, June 28- - July 2, 2011, pages 465 472. Omnipress, 2011.
[11] A. Filos, C. Lyle, Y. Gal, Levine, N. Jaques, and G. Farquhar. Psiphi-learning: Reinforcement learning with demonstrations using successor features and inverse temporal difference learning. In M. Meila and Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 3305-3317. PMLR, 2021.
[12) C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks, 2017.
10

---

[13] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML. 2018, Stockholmsmüsssnn Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1582-1591. PMLR, 2018.
[14] D. Ghosh, A. Gupta, A. Reddy, J. Fu, C. M. Devin, B. Eysenbach, and S. Levine. Learning to reach goals via iterated supervised learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[15] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint 1801.11000 2018.
[16] D. Hafner, T. P. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[17] N. Heess, G. Wayne, D. Silver, T. P. Lillicrap, T. Erez, and Y. Tassa. Learning continuous control policies by stochastic value gradients. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2944-2952, 2015.
[18] M. Hessel, H. Soyer, L. Espeholt, W. Czarnecki, S. Schmitt, and H. van Hasselt. Multi-task deep reinforcement learning with popart. In The Thirty-Thing AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27- February 1, 2019, pages 3796 3803. AAAI Press, 2019.
[19] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. In H. M. Wallach, H. Larochelle, A. Beygelzimer, d' Alchhé Buche E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 12498-12509, 2019.
[20] M. Janner, I. Mordatch, and S. Levine. Gamma-models: Generative temporal difference learning for infinite-horizon prediction. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information P Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[21] L. P. Kaelbling. Learning to achieve goals. In International Conference on Machine Learning. Citeseer, 1993.
[22] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. CoRR, 0bs/2104400212, 2021.
[23] V. R. Konda and J. N. Tsitsiklis. Actor-critic algorithms. In S. A. Solla, T. K. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 - December 4, 1999], pages 1008-1014. The MIT Press, 1999.
[24] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-leannin for offline reinforcement learning. CoRR, Abs/2906.04779, 2020.
[25] N. O. Lambert, K. S. J. Pister, and R. Calandra. Investigating compounding prediction errors in learned dynamics models. CoRR, abs/2203.09637,, 2022.
[26] L. Lehnert and M. L. Littman. Successor features combine elements of model-free and model- based reinforcement learning. J. Mach. Learn. Res., 21:196:1-196:53, 2020.
[27] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1333, 2016.
11

---

[28] S. Levine and V. Koltun. Learning complex neural network policies with trajectory optimization. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing. China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 829-834 JMLR.org 2014.
[29] Z. Li, J.-F. Ton, D. Oglic, and D. Sejdinovie. Towards a unified analysis of random Fourier features. In International Conference on Machine Learning, pages 3905-3914. PMLR, 2019.
[30] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In M. Balcan and K. Q Weinberger, editors, Proceedings of the 33nd International Conference on Ma- chine Learning, ICMI. 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1928-1977. JMLR.org, 2016.
[31] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning arXiv preprint arXiv: 2013.
[32] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Ried- miller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
[33] T. M. Moerland, J. Broekens, and C. M. Jonker. Model-based reinforcement learning: A survey, CoRR, abs/2006. 16712 12, 2020.
[34] O. Nachum, Y. Chow, B. Dai, and L. Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'ché- Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8 8-14, 2019, Vancouver. BC, Canada. pages 2315-2325, 2019.
[35] A. Nagabandi, G. Kahn, R. S. Fearing, and Levine. Neural network dynamics for model- based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, pages
IEEE, 2018.
[36] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: example-guide deep reinforcement learning of physics- based character skills. ACM Trans. Graph., 37(4):143, 2018.
[37] V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine. Skew-fit: State-covering supersupe reinforcement learning. arXiv preprint (203::9993.0300 2019.
[38] M. Posa and R. Tedrake. Direct trajectory optimization of rigid body dynamical systems through contact. In E. Frazzoli, T. Lozano-Pérez, N. Roy, and D. Rus, editors, Algorithmic Foundations of Robotics X- Proceedings of the Tenth Workshop on the Algorithmic Foundations of Robotics, WAFR 2012, MIT, Cambridge, Massachusetts, USA, June 13-15 2012, volume 86 of Springer Tructs in Advanced Robotics, pages 527-542. Springer, 2012.
[39] A. Rahimi and B. Recht. Random features for large-scale kemnel machines. Advances in Neural Information Processing Systems, 20, 2007.
[40] A. Rahimi and B. Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Advances in Neural Information Processing Systems, 21, 2008.
[41] A. Rudi and L.. Rosasco. Generalization properties of learning with random features. Advances in Neural Information Processing Systems, 30, 2017.
[42] O. Rybkin, C. Zhu, A. Nagabandi, K. Daniliddis, I. Mordatch, and S. Levine. Model-based reinforcement learning via latent-space collocation. In M. Meila and T. Zhang, editors, Proceed- ings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9201- PMLR, 2021.
[43] Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz, Trust region policy optimization. In ICML, 2015.
12

---

[44] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.
[45] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. P. Lillicrap, K. Simonyan, and D. Hassabis. Mastering chess and shogi by self-play with general reinforcement learning algorithm. CoRR, abs/1712.0188 2017.
[46] H. T. Suh, Tr. Pang, and R. Tedrake. Bundled gradients through contact via randomized smoothing. IEEE Robotics Autom. Lett., 7(2)4400-4407,, 2022.
[47] R. S. Sutton. Integrated architectures for learning, planning, and reacting based on approxi- mating dynamic programming. In B. W. Porter and R. J. Mooney, editors, Machine Learning, Proceedings of the Seventh International Conference on Machine Learning, Austin, Texas, USA, June 21-23 1990, pages 216-224. Morgan Kaufmann, 1990.
[48] R. S. Sutton, J. Modayil, M. Delp, T. Degris, P. M. Pilarski, A. White, and D. Precup. Horde: scalable real-time architecture for learning knowledge from unsupervised sensorimotor interac- tion. In L. Sonenberg, P. Stone, K. Tumer, and P. Yolum, editors, 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2011). I. Taipei, Taiwan, May 2-6, 2011, Volume 1-3, pages 761-768. IFAAMAS, 2011.
[49] Y. W. Teh, V. Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu. Distral: Robust multitask reinforcement learning. CoRR, abs/1 7ot.//477744 2017.
[50] A. Venkatraman, M. Hebert, and J. A. Bagnell. Improving multi-step prediction of learned time series models. In B. Bonet and S. Koenig, editors, Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence January 25-30, 2015, Austin, Texas, USA, pages 3024-3030. AAAI Press, 2015.
[51] C. J. C. H. Watkins and P. Dayan. Q-learning. In Machine Learning, pages 279-292, 1992.
[52] G. Williams, A. Aldrich, and E. A. Theodorou. Model predictive path integral control: From theory to paraliel computation. Journal of Guidance, Control, and Dynamics, 40(2):344-377, 2017.
[53] G. Williams, P. Drews, B. Goldfain, J. M. Rehg. and E. A. Theodorou. Aggressive driving with model predictive path integral control. In D. Kragic, A. Bicchi, and A. D. Luca, editors, 2016 IEEE International Conference Robotics and Automation, ICRA 2016, Stockholm, Sweden, May 16-21, 2016, pages 1433-1440. IEFE, 2016.
[54] R. J. Williams. Simple statistical gradient- algorithms for reinforreme learning. Mach. Learn., 8:229-256, 1992.
[55] C. Xiao, Y. Wu, C. Ma, D. Schuurmans, and M. Müller. Learning to combat compounding-error in model-based reinforcement learning. CORR, abs/1912.11 1206, 2019.
[56] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 1094-1100. PMLR, 2019.
[57] J. Zhang, J. T. Springenberg, J. Boedecker, and W. Burgard. Deep reinforcement learning with successor features for navigation across similar environments. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2017, Vancouver, BC, Canada, September 24-28, 2017, pages 2371 2378. IEEE, 2017.
[58] R. Zhang, B. Dai, L. Li, and D. Schuurmans. Gendice: Generalized offline estimation of stationary values. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[59] M. Zucker, N. D. Ratliff, A. D. Dragan, M. Pivtoraiko, M. Klingensmith, C. M. Dellin, J. A. Bagnell, and S. Srinivasa. CHOMP: covariant hamiltonian optimization for motion planning. Int. J. Robotics Res., 32(9-10): 10)--00))11641193 164- 2013.
13

---

Supplementary Materials for "Self-Supervised Reinforcement Learning that Transfers using Random Features"
A Algorithm Pseudocode
Algorithm Model-Free Transfer with Randomized Cumulants and Model-Predictive Control
1:
Input: Offline dataset 'D given in Section 2 distribution p over Idd, number of random features
K
2: Offline Training Phase: Randomly sample [Bx]RE[K] with 02 ~ p. and construct dataset
3:
Do = { (
aTH).
Σ
7 h-
p(sh h ah: 0k))
kE
mE(A(],E((K)
4: Fit random Q-basis functions ;k)),,)))) : S x A + RR for &s E [K] by minimizing the loss over the dataset Dø-
(vk) (UR)NE(K)
E argmin {or hejay
Σ
Σ
nh 16h ah:; 0z))
M
mE[M]]AE|K]
kE(H)
5: Online Planning Phase:
6: Fit the testing task's 's reward function r(:.) with linear regression on random features:
w² E aargmin
Σ
(r(s%) ag.
Σ
2 + Al|wo| w ax;; 0x))
MH
kE[H),ME|M]
kE[K]
where rss a.r Rop. .
7: Sample 81 140
8: for time index 1.... do
9: Randomly generate N sequences of actions {a",- ... , .iithee[N]
Find the best sequence such that
my E argmax
Σ
a dP
NE[N]
KE[K]
Execute " from the sequence n;, observe the new state 81+1 ((s,,
11: end for
B Implementation Details
Our implementation contains a few optional components that are omitted from main paper due to page limit. We note that RaMP's performance already surpasses the baselines without them. These components are introduced for the completeness our main algorithm and leads to better sample efficiency in different situations.
B.1 Infinite-Horizon Q-function Variant
While our setup in Section 2 uses shite-horizon Q" to approximate Qz, our method can also plan with an infinite-horizon Q function during the online phase. In this section, we describe one compatible way to learn an infinite-horizon Q-function while still enjoying the benefits of RaMP in the case with deterministic transition dynamics. We find this variant leads to continual improvement and thus better performance in few longer-horizon environments.
14

---

We first notice that an anfinite-horizan Q-function 1 can be decomposed into the discounted sum of an H step reward and a discounted value function under policy evaluated $t+
H
Qn(s', =
( V (8H+1)) +
Σ
r(se: ar) 81 8', = a¹)
a (x)) #1:1 (1,81)
t=1
20
where Vs((') Eu sc)
y r
81 =
Given a policy n, value function Vi parameterized by 0 can be learned via gradient descent and Monte-Carlo method:
-
- - (+v) (nr + V (s++1)) for sampled (Sex+1, ae, re) ~ Tm
where T trajectory rollouts collected with current policy x and I' is target network that gets updated by 0 with momentum.
Now consider our multi-step setup. Our multi-step Q-function also be written the sum of our H step approximation and discounted value function at 81411
20
(1 (s, 01.H) *t (-1 ((.ar))
re 11 =,01 -..... oo UH
= = (s, (s, 2I:H) + ( ($H+1),
where we note that the last line, there is no expectation over s4+1 since the transition dynamics is deterministic, and 81+1 is deterministically decided by ($1, 01:H).
Vanilla RaMP enables efficient estimation of Q, with novel reward function the cost of truncating the second term above with Q s Q# As we have shown in Section 4 planning with this finite- horizon Q-approximation would already lead to reasonable planning in most of the experiments.
We can go a step further and also estimate the second term so we can plan in an infinite-horizon setting. The main challenge is getting Va ($#+1) in multi-step setup, we don't 't explicitly predict #H+1- This, however, can be addressed casily by reparameterizing Vx (5 on an action sequence that leads to #4+1 +1+1J0 just like what we did for Q. We thus define multi-step value function
F (s, E.n-1 Vr (SH+z))
s, 0, ōΙ, ag = H
Then Q=(s, (1:11) = Q# a1.1)) ( + (s, a1.1). Under our deterministic transition dynamics, 81+1 is fully determined by (81, (.:H). so we can remove the expectation in the equation. We then rewrite the training objective V in terms F2 to learn F,:
0 - 0 - V |IV (8++)) (re4 + + (841+1))) for sampled ($:4 Oz4 F14H) H) Ta
becomes
-- 1- V [ am4+H-1) (re## + FFF (441))))44+)) for sampled (Se:e1, r(+H) TT.
where we parameterize multi-step value function Fg by Fe aT- So we can learn Fz with Monte- Carlo sampling and gradient descent just like what we do to learn V, in the single-step case above. Combined with Q we now have an estimation for the infinite-horizon Q-function Qr in a multi-step manner.
For planning. we do on policy learning by alternating between policy rollout and Q learning. As policy, MPC planner first uses the infinite-horizon (s, 01:H Q" (s. 01:H) + +n.1 F, (s, 01:H) to plan and collect rollouts. Then F, is trained with these on policy rollouts while Q" is also learned like vanilla RaMP via online least square. By incorporating this infinite-horizon variant, MPC planner can now plan with an infinite horizon during the online phase.
Infinite-horizon Q-uunctoon variant is used in all benchmarks in Figure
15

---

Reach
Reach Wall
D'Claw
Epicade Return
0.8
0.6
0.0
2
0-
0.2
0.2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
1.25
Environment Steps
1e4
Environment Steps
1e4
Environmentt Steps
le4
No Finetuning
Finetuning
Figure 5: Finetuning results on MetaWorld and D' D'Claw. Our method is able to continuously improve by finetuning the Q-basis networks during online training.
B.2 Online Psi Finetuning
While we freeze the Q-basis networks during online training in our setup in Section 2 we can also finetune the Q-basis networks to continuously improve our estimate of the Q-value. After performing reward regression for number of steps, we can finetune the Q-basis networks on online trajectories by fitting the predicted Q-values to the Monte-Carlo Q values and allowing the gradients to flow illustrate the effectiveness of finetuning, we train our algorithm on
through the regression weights. set of environments for 12800 steps, where we freeze the regression weights and begin finetuning after 6400 environment steps. As shown in Fig. 5 our method sees a noticeable performance increase with finetuning. We incorporate finetuning in our Hopper and D' O'Claw experiments, where we finetune psi network every 800 steps.
B.3 Planning With Uncertainty
Since the Q-basis networks are trained on an offline dataset, they can provide inaccurate Q estimates when online planning goes out of distribution. To encourage the agent stay within distribution, train an ensemble of Q-basis networks and augment the online planning objective with a disagreement penalty. Let (vm) mEEMM] denote the ensemble of Q-basis networks. The planning objective becomes
1i E argmaax NE[N]
1
Σ
whtm (St, C C :3 Var
(
Σ KE[K]
w wp (St, a 414tH-1i
M
me(ar)
[M[M] KE[A]
(B.1)
where is a hyperparameter controlling the weight of the penalty. We use B for all our experiments.
B.4 Model-Predictive Path Integral (MPPI)
Our method benefits from powerful planning algorithms such as model-predictte path integral (MPPI) (33] MPPI is an sampling-based trajectory optimization method which maintains a distribu- tion of action sequence initialized isotropic standard Gaussian. In each iteration, MPPI draws action sequences from the current distribution and computes their value using the learned Q-basis networks and online regression weights. It then updates the action distribution using weighted mean and standard deviation of the sampled trajectories, where the weights are computed as the softmax of the values multiplied by temperature parameter 7, We MPPI planner with y 10 for the D' O'Claw environment.
C Setup Details and Additional Experiments
In this section, we provide more details of the experiments, including more detailed setup and supplementary results.
16

---

C.1 Description of Environments
We describe the details of all used environments such as observation space, action space, reward, offline / online objectives, and dataset collection.
Meta-World All our Meta-Worle [56] domains share the standard Meta-Workd observation which includes gripper location, and object locations of all possible objects involved in the World benchmark. Although the observation space has 39 dimensions, each domain only uses one two objects so only 7 dimensions are changing any each domain we chose. For pixel observation variants of each domain, we concatenate two 84 84 RGB images from two views, with a resulting observation dimension of 42336. Each domain has a 4 dimensional action space, corresponding to the delta movement of end-effector in the workspace along with the delta movement of the gripper finger. Metaworld provides a set of 50 goal configurations for each domain. We collect offline dataset following the procedure described in Sec. C.2 The online objective is chosen to be a novel configuration that isn't any of the 50 offinn goal configurations. To create the privileged dataset, we choose 25 of the goal configurations as offline objectives. These chosen configurations are the furthest 25 goals from the online objective in the Euclidean distance. We evenly annotate the offline dataset with rewards for each of these goals to form a privileged dataset such that the online objective is out of the distribution of the offline objectives. For different configurations of the same domain, since object locations are in observation and the goal configuration isn't 't in it, the dynamics is the same. We now describe the objectives of all used meta- world domains, including those used in the appendix.
1. Reach across Wall The objective is to reach target across a wall. The location of the target is randomized for each goal configuration.
2. Open Door The objective is to open the door certain angle by putting the end-effector between the door handle and the door surface. For each configuration, the location of the door is randomized.
3. Turn on Faucet The objective is to turn on a faucet by rotating it along an axis. For each goal configuration, the location of the faucet is randomized.
4. Press Button The objective is to press button horizontally. The location of the button is randomized for each goal configuration.
Hopper Hopper is an environment with higher dimensional observation of 11 dimensions and an action space dimension of 3. Hopper is a locomotion environment that requires long-horizon reasoning since wrong action will make it fall down and touch the ground only after some steps. The objective of the original Hopper enviroment in OpenAl gym is to train it to run forward. To analyze modify the objective to a goal-conditioned variant
the performance of CQL and SF on Hopper, such that the agent is trained to follow a certain velocity specified by its goal. Similar modifications are common in meta reinforcement learning such as in [1]. We sample a set of 50 goal velocities between 0.5 and 0.75 as offline objectives to collect data in the same way as we did in Metaworld environment. To ensure good coverage of mate-actionn space, we collect data following set of 50 e-noised expert policies. For online transfer, we choose four domains with four different online objectives. All the variant domains of Hopper share the same dynamics.
1. Hopper Stand The objective is to stand still and upright. This online objective is on the boundary of the offline objectives' range.
2. Hopper Jump The objective is to jump to height of 1.5, without moving forward or backward. Such height is typically not achievable when running forward, so this objective drastically different from offline objectives.
3. Hopper Backward The objective is to run backward at a target speed of 0.5, which is in the opposite direction of the offline objective. Falling down to the ground is penalized.
4. Hopper Sprint The objective is to run forward at a speed of 1.5, which is out of the distribution of offline objectives ranging from 0 to 1.0. The reward function remains the same as that for offline objectives. Only the goal changes.
The 50 e-noised expert policies used to collect data contains 10 policies from Hopper Stand, 10 policies from Hopper Jump, 10 policies from Hopper Backward, 10 policies from Hopper Sprint and
17

---

10 policies from another hopper environment whose goal is running at random velocity between 0 and 1. We note the 50 policies doesn't provide any information about their corresponding rewards because the reward is only computed towards the different set of goal between 0.5 and 0.75.
D'Claw D' D'Claw environment is dexterous manipulation environment with 24 dimensions of observation space and 9 dimensions of action space. The 9 dimensions correspond to the 9 joints of a robot hand with fingers and 3 joints on each finger. The objective of the environment is to control the hand to rotate a tripod to a certain angle. Angular distance to the target angle is used as the offline objective. To increase the degree of freedom and make the environment more challenging, we allow the tripod to not only rotate but also translate freely on the 2d plane. The initial rotation and position of thetripod are randomized upon every episode. We collect the offline dataset in the same way as in meta-world, training on 50 offline objectives and using e-greddy to collect rollouts. At test time we choose a new offline objective angle and annotate the rewards of the privileged dataset in the same way we did for goal conditioned Metaworld environments.
Point Point is an analytical 2D goal-reaching environment with linear dynamics. The reward is defined as the distance to goal minus an action penalty. The offline objectives are negative distances to the randomly selected goals on the 2D plane. The online objectives are novel goals on the plane. Since we are not evaluating CQL and SF on this enviromment, we don't 't generate the privileged dataset for Point.
Point Perturbed Point Perturbed shares the same linear dynamics as Point, but features unsafe regions with negative rewards local maxima with small positive rewards. These perturbations represent out-of-distribution test objectives that cannot be well approximated by a low-dimensional feature vector. Note that the added perturbations make Point Perturbed no longer an instance of the Point environment with a different goal. Instead, it features online objectives that are completely in a different class from Point.
C.2 Training Details
For each domain, we first train 50 policies with SAC [15] Each policy is trained towards some offline objective of the domain described in Sec. C.I for 50000 steps. We then use e-greedy version of each trained policy to collect 32000 data points for each domain per offline objective. We choose 0.5. Such a procedure ensures the dataset has reasonable coverage of the entire state-action space. We note training these policies are fully optional, since RaMP only trajectories without rewards. Datasets collected via intrinsic rewards like curiosity would totally suffice. We also note that the environment used to train expert policies are not necessarily the same as environments used collect the data points. They should share the same dynamics yet the only the later provides reward information to our offline dataset. In particular, all metaworld data are collected via their original goal conditioned policies while hopper data are collected with a mixture of running and jumping policies in a different goal-velocity- conditit-eo environment as decribed in Sec. C. We choose the random feature dimension to be 2048. Each dimension in the random feature is extracted by feeding the state-action tuple to randomly initialized MLP with 2 hidden layers of size of 32. There are therefore 2048 independent, small random MLPs to extract Φ. All state-action tuples are projected to reward basis é with it.
During offline training phase, we ensemble 8 instances of MLP with 2 hidden layers of size 4096 and train & network following Sec. 3.2 We train e network with learning rate of 3 x 10-4 on the offline dataset for 4 epochs, with a 7 decay of 0.9 and batch size 128. We choose the horizon // be 16 for Meta- first do random exploration for 2500 steps to collect enough data points for linear regression. When doing linear regression, we always concatenate a bias dimension to t.. Since online least square makes recomputing w regression fast, we perform an update of weight vector every single step after initial random exploration is finished. In each MPC step, we randomly sample 1024 action sequences.
and Hopper environments and 10 for D'Claw. During online adaptation phase, we
We penalize the predicted rewards with the variance of predictions from all 8 ensembles following Sec. B3
18

---

1e2
Pixel Reach
1e2
Pixel Faucet Open
1e2
Pixel Door Open
5.0
Episade Retur
4.5
4.0
3.5
2
3.0
1-
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Environment Steps
1e4
Environment Steps
1e4
Environment Steps
1e4
RaMP (Ours)
Dreamer
Figure 6: Results on Metaworld with high-dimensional pixel observation. RaMP achieves comparable performance to Dreamer on Pixel Reach and Pixel Faucet Open but struggles to perform well on Pixel Door Open. The performance of RaMP on Pixel Faucet Open and Pixel Reach is very similar to its performance state observation environments.
C.3 Scaling to High-dimensions Pixel Observation
In Sec we evaluate RaMP's ability to scale to environments with high dimensional observations. In this section, we go step further by evaluating RaMP on visual domains, effectively increasing dimension of the observation space to 42336 as described in Sec. C.I We use a CNN encoder following the architecture in [32) followed by 2 layer MIP as the random feature network. Both CNN and MLP layers are randomly initialized. Action is projected and concatenated to the first layer of MLP the random feature would still be conditioned on both action and observation. We compare our method against the Dreamer [16), the state art model-
learning method optimized for pixel pre-train dreamer's dynamics branch with offline data before the online phase. As shown in Fig. 6 our method is able to achieve a similar level of performance Dreamer in two meta-wer world environments, Pixel Reach and Pixel Faucet Open. RaMP does not see a significant return drop from the variant of the environment with state observation. Given the scale up to pixel observation.
observations. Similar to MBPO,
efficacy of Dreamer, the result still shows RaMP's performance However, Dreamer is able to outperform RaMP significantly in an environment like Pixel Door Open. This is likely because random features capture the change in input space rather than reward space. Pixel observations can give random features a lot of noise while important semantic features may correspond
big change pixel space. We note that instead of using random convolution layers, we can use pre-trained encoders to achieve better semantic feature extraction and significantly improve the quality of random features. This beyond the scope of our work and we leave this for future works.
C.4 Additional Results on Hopper Sprint
Due to page limit, we omitted the plot for Hopper Sprint in Fig Here we provide additional results of RaMP and baselines for Fig. 7 The result is consistent with analysis in the main paper. RaMP matches or outperforms the baselines just as other Hopper variants. One major difference here is that CQL is performing well for HopperSprint. This is likely because the online objective of Hopper Sprint is the boundary of the offline objectives described in Sec. C. Given that offline objectives are running at target velocities, the CQL likely learns to not fall down even if the online objective is out of distribution. By not falling down alone, CQL is capable of maintaining good reward seen in this case.
1e3
Hopper Sprint
Episode Return
-2
1
2
3
Environment Steps
RaMP (Ours)
SF
CQL
C.5 Additional Ablations
Figure 7: Results on Hopper
Our method builds on the assumption that a linear combination of high dimensional random features can approximate arbitrary reward functions. In the following ablation experiments, we validate this assumption both quantitatively and qualitatively.
Sprint
19

---

4.1e1
Point
Point Perturbed
1e3
Metaworid Reach
1.0
Episode Return
2
0.6
0.4
-2
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Environment Steps
1e4
Environment Steps
1e4
Environment Steps
1e4
128
256
512
1024
2048
4096
Figure& 8: Learning curves for different random feature dimensions. Low-dimensional random features suffer from poor convergence performance, whereas high-dimensional random features experience slow adaptation.
Ground Truth Reward
basis_ dim 54
basis diin 256
basis dimi 1024
-0.5
-1.0
Figure 9:
Visualization of reward approximated with different numbers of random basis. As the number of random basis increases, the reward approximation becomes more accurate.
Effeet of different types of featurization We experiment with three choices of projections: random features parameterized by a deep neural network, random features parameterized by a gaussian matrix, and polynomial features of state and action up to the second order. We evaluate these choices on Point and MetaWorld Reach in Table 2 We see that a linear combination of NN- WN-paramettrized random features approximate the true reward well in all tasks. Polynomial features perform well on environments with simple rewards, but struggle as the reward becomes more complex, while Gaussian features rarely expressive enough.
Effect of random feature dimension We evaluate our method on Point, Point Perturbed, and Metaworld Reach using {128, 256, 512, 1024, 2048, 4096) random features. We summarize the final returns in Table. and plot the learning curves in Fig. g We find that performance degrades as the feature dimension gets lower because the random fea- tures are less capable of approximating the true reward. On the other hand, nigher-dimensional random features experience slower adaptation. In addition, provide visualizations of reward approximation in Fig. 9
Point
Reach
Random
34.0 + 1.0
820.8 + 142.0
Gaussian
-3.6 + 7.4
188.6 + 49.2
Polynomial
24.5 + 5.8
162.9 + 36.4
Table 2: Return for different features. Random fea- tures ahle to approximate the true reward well across domains. Polynomial features work in simple environ- ments but not scale to complex rewards. Gaussian features are unable to express the reward.
Scaling with state dimension In Table 4 we evaluate method analytical point environments with 2, 3, and state dimensions. All three environments feature distance-to-gon with an action penalty as the reward. We compute the return eπror stemming from linear regression well as the Q error stemming from both linear regression and function approximation. We see an increase both return error and Q error with higher state dimensions, but overall the approximation errors remain reasonably low.
Nonlinear approximation capability In Fig. 10 we visualize the truncated Q value obtained from linear combination of the random cumulants and compare the ground truth Q value approximated by Monte-Carh Carlo sampling. We perform the comparison in Point and Point Perturbed environments.
20

---

Table 3: Return as function of random feature dimension. Low dimensional random features are unable to approximate the true reward with linear regression. leading to degraded convergence performance.
Point
Point Perturbed
Metaworld Reach
128
30.46 + 1.62
29.61 + 2.55
765.38 + 129.98
256
31.53 = 1.45
31.09 + 0.04
806.12 104.77
512
33.02 + 1.10
33.13 + 2.09
824.56 = 121.52
1024
33.36 - 1.10
32.02 2.38
855.14 85.23
2048
34.01
+ 1.04
31.83 + 1.31
830.84 + 142.02
4096
33.31 + 1.15
32.05 + 1.11
802.45 94.47
Table 4: Approximation error with different state dimensions. As state dimension increases, approxi- mation eгror increases but remains in reasonable range.
Point 2D
Point 3D
Point 4D
Return Error
0.42 0.20
1.20 0.61
1.32 + 0,70
Q Error
0.32 + 0.05
0.51 + 0.10
0.61 + 0.12
Our method provides an accurate estimate of the Q value even in the face of out-of-distribution and highly nonlinear rewards.
In-distribution baselines Finally, we compare the performance of RaMP to baselines in-distribution online objectives. Our method designed to make assumptions about test objectives during policy transfer. As shown in Fig. 4 RaMP outperforms CQL and SF when the online objectives are of distri- bution. A natural question to ask is how things will change when the setting satisfies the assumptions of offline-online objective correlation. For example, in a 2D reaching environment, the training dataset may be annotated with either rewards corresponding to only goals on the right half goals covering the entire plane. When the online objective is to reach goal on the left half of the plane, will be out of distribution for the first case while being in distribution for the second. As shown in Fig. 11 when we curate the labeling process of the privileged dataset to satisfy in-distribution assumption, CQL and SF receive a significant performance boost, while the performance of our method and MBPO are unaffected as neither algorithm depends on the offline objectives. This serves foil the generalization of our method to out-of-distribution online objectives.
Predicted Values
75
Point
4.0
Point Perturbed
330
Figure 10: Visualization of true Q value and approximated Q value. Our method is able to approximate the Q value in the face of out-of- distribution and highly nonlinear rewards.
D Detailed Theoretical Results
In this section, provide the formal statement of the theoretical insights given in 3.4 and corresponding proofs.
D.1 Formal Statement
Theorem D.1. Suppose the offline data in are generated from some distribution p E ASS A), ie.. (slf ax -) and sht1 ~ T for all (m, h) E [M] x [H]. and inf p(s, a) = > 0.
(x,a)ESxA
Suppose 0g p(-) for all l [K], and
sup Axx
lo(s, a; 0) S R. for some K > 0 and
21

---

1e2
Button Press
1e3
Hopper
le2
D'Claw
Episade Retum
1.5
2
0
0.0
0.5
1.0
0.5
1.0
0.0
0.5
1.0
Environment Steps
1e4
Environment Steps
1e4
Environment Steps
1e4
RaMP (Ours)
MBPO
SF
SF In Dist
COL
COL In Dist
Figure 11: In-distribution results for Successor Features and CQL. Note that RaMP and MBPO are unaffected since they do not depend on the distribution of the offline objectives.
:()):0)) continuous. For some large enough := MH, letting À n-1/2 we have that K Oo(Vllog('* (r))). then with probability at least 5, for any given reward function R
/ [(r- (l.ff()))) (T)))))))) (llg(((/0))))
log(1/0)
(1²) 22 S 1 y
inf IEH
Σ
Vn
(s.0)ESXA)
E(
for any policy æ, where H to 3.1 and
::= = =-= = -0000000000000000000000000111111 (w000)/0(((01100(((((((()))))))) < oo), w" is the solution
(x,,,,w"))= : E
w (8n, uni 0k) -,
(D.I)
A=1
KE(K]
The proof of Theorem D.1 can be found in D.2 It shows that with large enough amount of random features, the Q-function any reward function It, under any policy π, can approximated accurately up to some inherent error related to the richness of the function class that the features can represent. Note that we here only state the results under some mild and basic assumptions from the random features literature, and are by no means tight. They be improved in various ways, for example, if the sampling distribution 0, p, can be data-dependent, and some stronger assumptions on the data and kernel function classes 41 29].
Corollary D.2. Suppose the assumptions in Theorem D.I hold, and additionally the kernel induced
function space H. is rich enough such that inf few E(T 0. Then, with horizon length H iew
( log( Riax/ /c) and M 6( ( episodes in dataset D, and with K = 52(1 2
random features, we have [| (w*) Qx I O(e) for any ग,, where for each (s, a). Q4 (s, a; w*)
a H -horizoo truncation of (D.I) which can be estimated from the offline dataset D.
The proof of Corollary D.2 can be found in $D.2 which specifies the number of random features, the horizon length per episode, and the number of episodes, in order to approximate Q, accurately by using the data in D. Note that the number of random features is not excessive and is polynomial in problem parameters. Combining Theorem D. and Corollary D.2 leads to the informal statement in Theorem 3.1
Next, we justify the use of multi-action Q-functions planning in deterministic transition environ- ment, which contains all the environments empirical results we have evaluated. Recall that for any given reward R, let Qn denote the Q-function under policy æ. Note that with a slight abuse of notation, Qx also be the multi-step Q-function (see definition in $2), and the meaning should be clear from the input, i.e., whether it is Q (8, a) or Q=
Theorem D.3. Let IT be some subclass of Markov stationary policies, i.e., for any E II. S - (AA). Suppose the transition dynamics T is deterministic. For any given reward R, denote the H H-step policy obtained from H H-step open-loop policy improvement over II as T S + A
22

---

defined as
(s) E
argmax max Qr a1, ,0HH) (on.,,au)eea nell
for all E S. Finally, define the value-function under TiH V (8) : Qui, (8, (s)), where Qry, (s, 01.H) is the fixed-point of the Bellman operator TH, defined in (D.8). Then, we have that for s E
V (s) 2 max max Qx a1,'" ,. 2 max max Qr FEll
TEll
D.2 Deferred Proofs
D.2.1 Proof of Theorem D.1
The proof relies the result of generalization guarantees of learning from random features, with squared loss. Note that one cannot use the results in [40], which dealt with Lipschitz loss function of the form c(t', y) = c(y y). This does not include the squared loss used in our experiments. Instead, we resort to the results in [41], which also yield better statistical rate. For the sake of completeness, we re-state the abridged version of key result therein, Theorem in 41], as follows.
Lemma D.4. Suppose that K is kernel with integral representation ((2,z') Ja (zx w)b(a' w)dp(w), where (2,p) probability space and e x Ω + I, where X a separable space. Suppose e is continuous and [ w)l s with x E [1, +oo) almost surely, and |y| 6 almost surely. Define the expected risk:
E(f) = / (f(=)) 2
where is the distribution where the data samples (I,, y:) Define the solution to kernel ridge regression with M random features as
(x) (x) úx.M, with Tx.M = SM -
(D.2)
where (x)) (1(1,, W1), (,, w2), ba wx)) /) /MM, w; drawn i.i.d. from p(-), 7 (y1.-.. 1/2 Th. == ( ... 6M (zxx))n²z // Then, suppose 2 X = 1/n
and the number of features M 2 aovTllo(118e 2 v we have that with probability at least 1 5.
C1 log (18/5)
E(fAM) - min E(f)
<
JEH
Vn
where no, Co. Cj are absolute constants, H the reproducing kernel Hilbert space corresponding to the kernel K.
We then apply Lemma D.4 with (x,y) in the lemma being replaced by ((s, a), r(,,)))))))) p(z, y), P, x, w, A in the lemma being replaced by a(s.a) R P, as, 0, K, À in Note that Lemma requires the space X to be separable, and our finite space S x A satisfies; it re- quires lyl bounded, and our reward is absolutely bounded by Rmax. and thus also satisfies. We hence obtain that with probability at least - 5, if the number of random features K 2 MiVn log/lool( (n)) with n HM, then
(esa))p(s)) (T- Σ
< inf E(f) 0 (0 log(1 1/0) fEH
wiols, aio(s,a;(s)) KE|K]
(D.3)
Fn
where we note that w² -wf
(wf,.... ... . is the solution to 3.1 and the E(f) here is defined in
Theorem D.I
For any policy æ for the MDP, let Q. denote the Q-function under policy and the actual reward function distribution R, and Qa (w") denote the -unectio under the estimated reward using random
23

---

features:
(n( E Σ
(sh:
On: w0)
81 8, 01
where a(s,a;"")
(D.4)
kE[K]
By Bellman equation, have that for each (s,
(x((s.)) Qn a; w²)) || /
rdR rdR
(0 ss)
F(s,a:w")
s'
< $ rdd - F(s,a; + 0 -
Taking sup over s, a and organizing the terms, we have
Qxx - Qx("))
| S ST- 1 -uu
( (rs,
s,a
Σ
(r) f(s a())
(D.5)
Σ
(r) -(s, p(s,a)
(D.6)
V
L
(1 -1-7) /P V (((sa)rplsaa
(frdRRa(t)-
(D.7)
where (D.5) uses that 11. S || lla for dimee--dimanionna vectors, D.6) uses the definition of p. Further, by Jensen's inequality, for each (s, a)
(ʃ (frdRaaatt F(s, ((rdB..())-(ss.:w""))
< EE F(s,
which, combined with (D.7) and (D.3), gives that
(f (rdReallt) f(s, as ux)))
Qr - -x0) I (1 ") V B(sa)rotssssonand Rmal))
0 (log(1/6) vn
inf E(f) + 0 SEH
(1 (1-7)v V
which completes the proof.
D.2.2 Proof of Corollary D.2
First, that with H O log( R.ma.. /(1(1-))1)) ensures that @! (w+) - (w)) O(e), which can be obtained by the boundedness of r(s, a) by Rmaxx and the fact that
1-7
log( Rinax /ewna//ee(11111- ((((-)))))
H Rmax = (1- - (1- 1-
(((-)))) (1)) Rmax
Rmaxx
1- <
1-7
Furthermore, since Theorem D.1
requires n = HM 5 (1))))) to make sure 11@m(w)) 1x < Combining these facts yields the desired result.

---

D.2.3 Proof of Theorem 3.2
Define
Qn (8,01, aH) max Q, (s, 01, ,az), and Q Qmn"((sa)) == max (s,
We also define the Bellman operator under the open-loop policy TH as follows: for any Q
R
||
TH (1, .ax) E
Σ (-s,,sh, ( (( 81 01:1 he|H]
(D.8)
Note that TH. is contracting operator, and we denote the fixed point of the operator as QH. R|S|x|A41 which is the Q-value function under open-loop policy THH By definition, we also know that the state-value function under TH. VH,*a = -HHm (8, THH (+)), i.., by applying the open-loop policy TH to the actions a1; Q 11 F (s, a1.H).
Note that
TH (Q (8, " a.H)
Σ h- r(s, + " max ( ((H+1)) 81 8, 01:1 NE(M
E
Aー - an) + +
max
QH (SH11HHH,,
a1.H
(D.9)
an 68412H
he(H]
E
1 and + Q=(59411
8, a1:7
(D.10)
an412n
ME(M)
E
- and Q7 ((((()))))))
81 8, 01:H
(D.11)
NE(H]]
= Q+ 01, aH): for any E II, where (D9p uses the definition of THe (.10) is due to the definition of Q and
(D.12)
I.III) is by the maxan and (D.12) is by definition. Since (D122) holds for any π II. by the monotonicity of TH,mp we have
QH (8, lim ** (1.H) 2 Qmax (s,) max Qx a1.H).
(D.13)
Notice that for all by applying (s) on both sides of (.13).
VH, (s) Q Qu,,,, (s, (s)) Z max Q* (s)) HEll
max max Q~ (s, a1:H).
(D.14)
and
Further, due to the multi-step maximization, we have
max max Qx a,,... aa)) max max 011
which, combined with (D.14), completes the proof.
E
Complexity Analysis
The space complexity of RaMP is primarily determined by the number of random features that are needed. As we describe in Corollary D.2 we require K = s)(( - -2--2) random features to achieve e-order error between the estimated and true Q-function. As the required approximation error decreases, the space needed to store all the features grows sublinearly. Note that this result is for any given reward function (including the target one) under any policy π, but not tied to a specific one (under the realizability assumption of the reward function).
On the other hand, the space complexity the classical successor feature method relatively fixed in the above sense: the dimension of the feature is fixed and does not change with the accuracy of approximating the optimal Q-function for the target task. However, more restricted: it is only for the optimal Q-function of the specific target reward, and also depends on the distance between the previously seen and the target reward functions. Hence, the two space complexities are not necessarily comparable.
resulting guarantee is also
25

---

Relationship to Existing Work
We briefly connect our proposed algorithm prior work.
Successor features for transfer in RL: While successor features S have shown the ability to transfer across problems in RL, the two key differences in our framework are (1) using random features rather than requiring learned or pre- pre-provided features and (2) training multi-action Q functions rather than typical Q.. These two changes allow transfer to happen across a broader class of reward functions and not simply be restricted to the policy cover experienced at training time.
Model-based RL:
Our work is connected model based RL in that disentangles dynamies and rewards, but is crucially different in that it doesn't 't model one-step evolution of state but rather long term accumulation of random features. This trades off compounding error for generalization error.
Our work is connected to methods for model-free RL in that it also models set of Q-functions, but importantly this doesn't 't correspond to particular reward, but rather random features of state. By doing we are able to adapt to arbitrary rewards at test-time, rather than being tied a particular reward function.
Model-free RL:
26


Reviews:
[{"review_id": "t1wwE1vTys", "reviewer": "Reviewer_W1dV", "summary": "The paper proposes a Random Features for Model-Free Planning (RaMP) algorithm to solve the problem of learning generalist agents that are able to transfer across platform where the environment dynamics are shared, but reward function is changing. The proposed algorithm leverages diverse unlabeled offline data to learn models of long horizon dynamics behavior, while being able to naturally transfer across tasks with different reward function. The authors evaluated RaMP on number of simulation based robotic manipulation and locomotions tasks.", "questions": "Please address concerns mentioned in the weakness section.", "limitations": "mentioned in the paper.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The problem addressed in the paper is very important problem for RL in robotics/robot learning real-life tasks.\n- The proposed method is very interesting and seems relevant to the research community.", "weaknesses": "- The paper is not well-written and very hard to follow. For example, Line 13-15 and Line 59-61 is very hard to follow.\n- The paper attempts to solve an interesting problem and has strong results in simulation based robotics task. Since, the motivation of the paper draws from real-robotic tasks, it would be good to see how this method performs on real-robotic tasks. I understand authors have mentioned in future work but to understand complete effectiveness of the proposed method, it seems critical. \n-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We respond to your comments below.\n\n> The paper is not well-written and very hard to follow. For example, Line 13-15 and Line 59-61 is very hard to follow.\n\nThanks for the comment. Line 13-15 simply means our method can be trained “without” reward labels, but on the other hand enjoy the benefit of being able to be quickly deployed to new tasks. Line 59-61 simply means that as long as the number of random features being used is large enough, we can estimate any test-time Q-function by a linear combination of the Q-basis functions. We will improve the writing in our updated version. \n\n>The paper attempts to solve an interesting problem and has strong results in simulation based robotics tasks. Since, the motivation of the paper draws from real-robotic tasks, it would be good to see how this method performs on real-robotic tasks. I understand authors have mentioned in future work but to understand the complete effectiveness of the proposed method, it seems critical.”\n\nThanks for the great suggestion. We agree that applying our method to real-robotic tasks would be a big plus. With the timeline constraints, we may not be able to finish such real-robotic experiments during the rebuttal phase. We will add them in the next version of our paper. We would like to emphasize that we have run our method across 8 different problem domains, and provide theoretical backing of the proposed algorithm. Hopefully, this provides convincing evidence for the reviewers of the empirical efficacy of our proposed method. \n"}, {"review_id": "QVWwswiLA0", "reviewer": "Reviewer_gqpD", "summary": "The paper tackles the problem of \n\nIn the training phase, they create $H$ randomly initialized neural networks that serve as reward functions during the purpose of pretraining. The Q-function learned online is a linear combination of these random features: $w^T\\sum_{h\\in [H]}\\phi(s_h,a_h)$.", "questions": "I'm a little confused by Figure 2. The way that the random features and Q-basis are drawn makes it look like each $\\phi_h$ is one of the basis vectors that together compose $Q-basis$. Everywhere in the text implies that each index of Q-basis is a different function parameterized by a different $\\theta_k$ and together the $\\theta_k$ make up the basis vector. Is that correct? If so I would recommend modifying Figure 2 left to include a reference to k.\n\nThe number of environment steps in the online phase seems really small. Why was 3000 chosen and about how many trajectories is it? E.g., in the IQL fine-tuning experiments they use ~1M environment steps.\n\nWhy did the authors not include D4RL data in their experimental evaluations?", "limitations": "N/A", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "This is a beautifully written paper. The motivation is crisp and the proposed method is very well grounded theoretically. Results across a broad range of tasks are presented. On originality, clarity, and method quality, I rate this paper highly.", "weaknesses": "The paper would benefit from more baselines: for example doing something like  IQL with a reward of 1 at the end of each trajectory during offline training. Less importantly, I also think IQL would be a more natural oracle than CQL.  It performs better than D4RL and directly evaluates online fine-tuning. In fact, the IQL paper shows that CQL can sometimes collapse during online fine-tuning.\n\nI also don't quite understand the choice of offline dataset: from the appendix, it sounds like the policies that generate the offline dataset are quite \"expert.\" It would be interesting to see how well the model performs if the offline dataset was instead modified from the D4RL benchmark, which includes different levels of expert policies. I expect high quality data to benefit any self-supervised method, but the claims of the paper are not limited to offline learning on expert demonstrations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We respond to your comments below.\n\n> The paper would benefit from more baselines: for example doing something like IQL with a reward of 1 at the end of each trajectory during offline training. \n\nThanks for the great suggestion. We have added new experimental results by comparing with IQL with the reward scheme that you suggested (see Fig. 1). This is akin to the CQL baseline in the original paper, but replacing the base algorithm CQL with the newer algorithm IQL. As expected, IQL performs less favorably compared to our approach, as CQL did. We will add the new experiments in our final version. The important point to note here is that model-free offline RL methods like CQL/IQL are inherently tied to the reward, while RaMP is not tied to the rewards but effectively models the system dynamics. This allows it to easily transfer to new rewards, in situations where algorithms like CQL/IQL will struggle to transfer since they are inherently tied to the reward. \n\n> I also don't quite understand the choice of offline dataset: from the appendix, it sounds like the policies that generate the offline dataset are quite \"expert.\" \n\nWe would like to clarify the dataset setup and present new results on D4RL. We use noised expert policy to collect trajectory with very aggressive epsilon noise (>= 0.5) so the trajectories rarely reach high reward. In addition, we don’t assume the dataset contains rewards so the test time objective can be quite challenging for the neural network to learn. In Fig. 4 of the supplementary pdf, we have added a D4RL experiment where we outperform CQL. The new experiment shows that our comparison isn’t due to how our data is collected. The important point is that the datasets must have coverage, as is typical in most offline RL algorithms, but the datasets can be very mixed with both optimal and suboptimal behavior. This is in contrast to imitation learning datasets that only have expert data. \n\n> It would be interesting to see how well the model performs if the offline dataset was instead modified from the D4RL benchmark, which includes different levels of expert policies. \n\nThanks for the great suggestion. We have added the use of D4RL in our new experiments, see Fig 4. The original reason why we didn’t benchmark in D4RL is that we want to develop a method that doesn’t assume train/test time objective correlation, and thus want to test on out-of-distribution goals or rewards. Most of the D4RL environments are not goal-conditioned and have the same reward function during the offline/online phase. This is drastically different from our reward-agnostic setting. In our new experiment, we chose only few goal-conditioned environments in D4RL and the results illustrate our performance isn’t due to our customized data but the algorithm itself.\n\n> I'm a little confused by Figure 2. The way that the random features and Q-basis are drawn makes it look like each\n… Is that correct? If so I would recommend modifying Figure 2 left to include a reference to k.\n\nSorry for the confusion. It is correct that the basis used to estimate Q is different from (but the accumulation of) the basis used to estimate the reward function, which is $\\psi$ instead of $\\phi$. For the features $\\phi$ and $\\psi$, their “parameterizations” are different also – the former is parameterized by $\\theta$, while the latter is parameterized by $\\nu$ (See our Sec. 3.2). We will make the clarifications in our updated version.\n\n> The number of environment steps in the online phase seems really small. Why was 3000 chosen and about how many trajectories is it? E.g., in the IQL fine-tuning experiments they use ~1M environment steps.\n\nOne of the main baselines, Successor Features, is extremely slow to run in terms of clock time. The 32000 step is a hyper-parameter which we choose by observing the number of environment steps needed to achieve success/converge across environments. Since planning backbone isn’t our main focus, we used a computationally inefficient version of random shooting MPC for both Successor Features and our method. Random projection for the successor feature baseline using many independent MLPs instead of one big MLP also made it slow, not to mention we have to copy these networks multiple times for ensemble. Official MBPO implementation is also very slow with ensembles so it’s also extremely difficult to train ~1M steps.\n\nPlease feel free to let us know if there are any other comments that may help reevaluate our paper.\n"}, {"review_id": "y6TzwmGFV4", "reviewer": "Reviewer_1b4X", "summary": "The paper proposes an approach for unsupervised pre-training of RL agents, ie pre-training on offline agent experience without rewards. The approach generates a set of random reward functions and then learns a successor representation of the state-action space for predicting cumulatives of these rewards on fixed-horizon trajectories from the pre-training data. At test time, it uses a small reward-annotated dataset to learn a linear mapping from the pre-trained successor representation to the target reward. Then it extracts a policy by greedily maximizing the resulting estimates of future cumulative rewards. The paper demonstrates that this allows for faster finetuning than model-based RL since only a linear mapping needs to be learned on the target task data instead of the full Q-function. In contrast to prior work on successor features it does not assume access to given features or a reward-annotated training set to learn features, but instead uses random reward projections.", "questions": "N/A", "limitations": "Limitations on image-based environments are demonstrated in the appendix but should be discussed more prominently in the main paper (see weaknesses (B)).", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- the problem formulation is impactful: pre-training with reward-free data has great potential for generalist RL agents\n\n- the paper is well-written and easy to follow, it clearly outlines the problem formulation, explains the method and provides theoretical justifications\n\n- the experimental evaluation seems comprehensive, with only minor experiments missing (see below): the paper compares to a representative set of baselines on multiple environments, including image-based control (though there with limited success) and also performs ablations of the key elements of the method\n\n", "weaknesses": "(A) **Relation to prior work not sufficiently explained**: the proposed approach heavily builds on top of prior work on success features for RL, yet the current submission only mentions this relation in a half-paragraph in the main paper and adds a slightly more detailed discussion at the very end of the appendix. It would be helpful to more clearly introduce this most relevant prior work and explain the deltas, so it is easier for readers to understand the main novel components. Concretely, one could add a preliminaries section that summarizes the idea of successor features and then point out the two main novelties in the proposed approach: random reward projections to be independent of training tasks and action chunking to be independent of training policies.\n\n(B) **Random reward features don't scale well to images**: the main difference to prior successor feature works is that the proposed approach avoids the assumption of pre-defined features or given pre-training task distributions by using random reward projections. The downside of this is that the algorithm cannot \"discard\" any information and struggles on high-dimensional inputs like images, where predicting random rewards over pixel inputs is complex and lacks semantic meaning. I agree with the authors that this can potentially be mitigated with pre-trained representations, but since removing the assumption of pre-defined features is one of the main selling points of this work, this tradeoff should be discussed more prominently in the main paper (currently only in appendix C.3) and the corresponding image-based results should be included in the main text. It would also be helpful to show the successor feature baseline results on the image-based domain.\n\n(C) **Missing Ablation**: The other introduced novelty is the use of action chunking, ie conditioning the Q-function on a sequence of H actions instead of on a single action. However, this choice is not ablated. It would be good to include versions of the proposed approach without action chunking (ie H = 1) and with different horizon lengths to see the benefit of chunking.\n\n(D) **Missing baseline**: The paper compares to goal-conditioned offline RL (CQL) but with a pre-specified selection of goals. Prior work has instead proposed to use goal-conditioned offline RL on randomly sampled goal states from an offline dataset for pre-training [1]. This has two benefits: (1) it does not require pre-defined goals / tasks, fully matching the assumptions of the proposed approach, (2) because of that it may be more robust to test time adaptation to new tasks. Thus, it would be good to add comparison to Actionable Models on all environments.\n\n**Minor Point**:\n\n(E) **MPC is misnomer**: The paper refers to the downstream policy extraction as \"model predictive control\" and \"planning\" (see eg Fig 2). However it does not perform prediction or use a model. Instead it performs greedy policy extraction by maximizing the Q-function, but uses action chunking (ie N-step action inputs). Thus I would call this step \"greedy policy extraction with action chunking\" instead.\n\n[1] Actionable Models, Chebotar et al. 2021\n\n\n## Summary of Review\n\nOverall I like the paper and am happy to accept it if the authors can more clearly mention the connections to prior work on successor features, discuss tradeoffs for image-based domains and add the suggested ablation + baselines. The idea of using random reward projections for successor feature learning is novel to my knowledge, but I am not 100% certain and will thus assign lower confidence to my review.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We respond to your comments below.\n\n> Relation to prior work not sufficiently explained.\n\nThank you for the great suggestion. We will add a preliminary section to summarize the idea of successor features, and emphasize our novelties in the section, in our final version of the paper. This would indeed make our contributions easier and more outstanding for readers to understand. You’ve hit the nail on the head with the key elements - random features rather than require pre-known successor features, and getting rid of policy dependence by doing open loop Q modeling wrt action sequences rather than having an implicit policy dependence. These are crucial because it allows us to learn from offline datasets without reward labels, and it allows us to learn Q-functions that transfer significantly better than techniques like generalized policy iteration (GPI). GPI can simply take the piecewise max between many optimal policies, while RaMP can optimize for any policy that is within the data support. We will make this significantly more clear in the document. \n \n> Random reward features don't scale well to images.\n\nThanks for the insightful comment. While RaMP can work directly from images using random convolutional features, as we have shown in Appendix C.5, this may perform more effectively using pretrained image features rather than using completely random convolutional features.  \n\nWhile one of our motivations is to *remove* the pre-defined features, what we meant by using *pre-trained* features, as a remedy to the scalability issue, is that we will first *randomly* project onto some feature space of the images, e.g., the latent space pre-trained over standard datasets as ImageNet, and then do random MLP features building on these pretrained features (as in our state based experiments). This is a fairly standard approach to reduce the dimension to consider, when dealing with images and still allows there to be random features, but now building on top of pretrained image features (not directly using those image features). We will make this point more explicit, and also move the related discussions and image-based results from the appendix to the main paper. \n\n> Missing Ablation.\n\nThanks for the suggestion. We ablated the horizon earlier under the finite horizon variant by shortening the horizon by half and four times. We found that longer-horizon environments like hopper would suffer slightly while our method retains similar performance on meta-world environments. Unfortunately, we ran out of time benchmarking enough seeds for this ablation under our current infinite horizon variant. We appreciate the suggestion and will add final results to the paper in the future.\n\n> Missing baseline - Actionable models\n\nSince source code is not available, we were not able to run this baseline in the short rebuttal period but we will certainly run this and include it in the final version. We would like to note that while actionable models are restricted to goal conditioned problems, RaMP can adapt to any reward function, even non goal reaching ones. \n\n> Minor point – MPC is a misnomer.\n\nThanks for the suggestion. The step uses “prediction” in the sense that the Q-function we used for optimization is an “accumulated” reward over $H$ steps, and it belongs to the MPC framework as we make “optimization” over a window of $H$ steps, but only *implement* the first step, and then interact with the environment before “re-optimize” the sequence of actions again. However, we certainly acknowledge the confusion here and we will rename the method accordingly. \n\nPlease feel free to let us know if there are any other questions.\n"}, {"review_id": "ABa48rGF1m", "reviewer": "Reviewer_BJ94", "summary": "The paper introduces RaMP, an approach for fast adaptation to unseen reward functions given offline data collected with arbitrary behavior policies under the same dynamics. RaMP learns a set of basis multi-step Q-functions, each corresponding to a random reward defined as the accumulation of random state action features. For online adaption to new rewards, the new reward/Q function is then identified using linear regression given the basis functions and used for control via MPC. \n\n**Edit After Rebuttal**:\n\nRaised score from 3 to 5, see comment below for details", "questions": "- I am a bit puzzled by the meta-world return curves: If I recall correctly successful execution in most of these tasks corresponds to an episode return of ~ 4,000. However, all presented algorithms achieve max. 1,000 (on \"Reach wall\") and considerably less in the 3 other tasks. Is there some normalization here? I would prefer to (additionally) see the success rates for these tasks to improve the interpretability of the results. \n- What exactly is indicated by the error bars? How statistically significant are these results given that it's only 4 seeds? \n- Are the \"implementation details\" (Appendix B) also used for the baselines? In particular: Additional value function for infinite horizon (B.1) for PeTS, online adaptation (B.2, but adapting the dynamics) for PeTS and MBPO, Planning With Uncertainty (B.3) for PeTS and MBPO, and MPPI (B.4) for PeTS.\n\n- What are the standard \"coverage and sampling assumptions\" in theorem 3.1? How realistic are those in praxis? What does the theorem add, compared with standard random feature regression results? ", "limitations": "I believe all limitations have been addressed, although some only in the supplement (infinite horizon problematic with multi-step q function), and I believe the paper would benefit from moving this also to the main part.\n ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "RaMPs' key idea, learning \"random feature rewards\" which are later combined, is novel and interesting as it allows offline pertaining without \"reward labels\" and provides an easy and efficient adaptation mechanism. The paper addresses a significant problem, the generalization to novel rewards under little to no assumptions on the knowledge of the current scenario and previously seen rewards, and is mostly clearly written and easy to follow. ", "weaknesses": "The paper's main weakness is its experimental evaluation, which I think is in its current form insufficient to validate the paper's claims or allow assessment of the method's potential for several reasons: \n\n- The algorithm described in the main paper seems to be improved by several \"implementation details\" (c.f. Appendix B). While the paper states that \"RaMP’s performance already surpasses the baselines even without them\", I believe a rigorous analysis of their impact would be necessary to assess the method's potential and workings.\n-  Baselines: none of the considered model-based RL methods (PeTS, MBPO, Dreamer) was designed to work with dynamics models trained on **offline** data. For a fairer comparison, methods designed to work with such dynamics models (e.g.  MOPO[1]) should be considered. \n- While RaMP adapts very fast in the considered environments, its final performance is often lower than that of MBPO and/or PeTS, in particular for the Hopper and D'Claw experiments. Further discussion and analysis in the paper would allow a better understanding of RaMPs limitations.\n- Compared with most recent work in RL and/or offline RL the paper only uses a small set (8) of relatively simple environments - the maximal state dimension is 39 (meta world, where only a subset of these is relevant for the considered tasks) and maximal action dimension is 9. \n- Higher dimensional observations (pixels) are used for some experiments in the supplement but the results are again inconclusive or seem to favor the baseline (Dreamer-V1, which is to the best of my knowledge not SOTA on pixel metaworld [2]) \n- See questions below. \n\nWhile the majority of the paper is clearly written and understandable, this is, in my opinion, not the case for section 3.4. Here the assumptions, their realisticness, and practical consequences of theorem 3.1 could be stated much more clearly \n\n[1] MOPO: Model-based Offline Policy Optimization, Yu et al 2020\n\n[2] Masked World Models for Visual Control, Seo et al 2022\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments.\n\n> The algorithm described in the main paper seems to be improved by several \"implementation details\" ... I believe a rigorous analysis of their impact would be necessary to assess the method's potential and workings.\n\nThanks for the suggestion. We will move more details from Appendix B back to the main paper, and make the statement more rigorously by  provide ablation experiments addition to those in appendix to validate the importance of each design choice. \n\n> Baselines: none of the considered model-based RL methods (PeTS, MBPO, Dreamer) was designed to work with dynamics models trained on offline data. For a fairer comparison, methods designed to work with such dynamics models (e.g. MOPO[1]) should be considered.\n\nThanks for bringing up MOPO. We’d like to clarify that  MOPO isn’t directly applicable here while also providing results of adopted MOPO by providing it with privileged information.  MOPO adds a penalization term to reward based on ensemble disagreement during policy training while our offline training data contains NO reward. Despite this, we adopt MOPO in a way similar to how we adopt CQL in figure 1 in rebuttal pdf. Despite privileged information, we observed that MOPO will suffer from similar problems that goal-conditioned CQL faces.\n\n> While RaMP adapts very fast in the considered environments, its final performance is often lower than that of MBPO and/or PeTS ... Further discussion and analysis in the paper would allow a better understanding of RaMPs limitations.\n\nThank you for the suggestion. We will add relevant discussion and analysis about this behavior in our final version. These differences in the 3 environments are due to the intrinsic problem of random-shooting MPC compared to policy based-methods. We recognize this limitation and would address accordingly in final version\n\n> Compared with most recent work in RL and/or offline RL the paper only uses a small set (8) of relatively simple environments ... maximal action dimension is 9.\n\nIf appendix is included, our paper has a total of 15 environments before the rebuttal. We also just added some D4RL results in figure 4 of rebuttal pdf. We note that we are NOT under an offline-reinforcement learning setup so suitable benchmarks aren't directly applicable here. We note a lot of highly cited works under similar setting like successor features only has 2 environments with much lower action dimension. \n\n> Higher dimensional observations (pixels) are used for some experiments in the supplement but the results are again inconclusive or seem to favor the baseline (Dreamer-V1, which is to the best of my knowledge not SOTA on pixel metaworld [2])\n\nWe present the pixel observation result mainly to illustrate that random projection also works on higher dimensional data and convolution networks rather than presenting our method as the state-of-the-art in RL from pixels. We will revise the writing and clarify the purpose of the experiment there.\n\n> If I recall correctly successful execution in most of these tasks corresponds to an episode return of ~ 4,000. However, all presented algorithms achieve max. 1,000 (on \"Reach wall\") and considerably less in the 3 other tasks.\n\nMetaworld is an environment which doesn’t explicitly provide a “done” signal for the environment. It didn’t provide a max episode length either so the benchmark with metaworld needs a manually specified max episode length. We chose a short episode length here since the four metaworld tasks don't need excessively many steps as in the original paper.\n\n> What exactly is indicated by the error bars? How statistically significant are these results given that it's only 4 seeds?\n\nThe error bars here are standard errors. We deem 4 seeds to be statistically significant since, unlike online reinforcement learning, we have a static dataset which majority of the training steps are trained on. This makes the models already very stable before the online adaptation phase and also avoids the variance from blind exploration. \n\n> Are the \"implementation details\" (Appendix B) also used for the baselines? In particular: Additional value function for infinite horizon (B.1) for PeTS, online adaptation (B.2, but adapting the dynamics) for PeTS and MBPO, Planning With Uncertainty (B.3) for PeTS and MBPO, and MPPI (B.4) for PeTS.\n\nYes, all baseline results are with online adaptation. For Planning With Uncertainty, we have a model ensemble for MBPO and SF baseline and a regularized version, MOPO in rebuttal pdf figure 1.  \n\n> What are the standard \"coverage and sampling assumptions\" in theorem 3.1? How realistic are those in praxis? What does the theorem add, compared with standard random feature regression results?\n\nWe have stated in the theorem that it is an “informal” version of the full result, due to space constraints. We will move the full version back to the main paper. The coverage and sampling assumptions, as we stated formally in Theorem D.1, means that the state-action pair is sampled from some offline data distribution that has coverage of the state-action space (diverse enough), and this is related to the standard “all policy concentrability” assumption in the offline RL theory literature, e.g., [1,2]. It is necessary for many offline RL approaches [3], and is important for our approach to be able to cover *any* new reward functions at test time. Compared to the random feature literature, we additionally need to analyze the propagation of the reward function approximation error to value function approximation (due to sequential decision-making), to adapt the loss used in the literature to that in our setting, and to analyze the effect of multi-step optimization over the Q-functions. We will make these points more explicit.\n\n\n[1] Finite-time bounds for fitted value iteration\n\n[2] Approximate policy iteration schemes: A comparison\n\n[3] Information-theoretic considerations in batch reinforcement learning\n"}, {"review_id": "iKstgkDI2Q", "reviewer": "Reviewer_8qBK", "summary": "The authors propose a self-supervised method to learn approximate multi-step Q-estimates by leveraging random features as bases for a reward function in the target task. This approach addresses some limitations of model-based and model-free RL. In particular, model-based RL typically suffers from compounding error of predicting future states from predicted states. On the other hand, model-free RL tends to suffer from instability when transferring knowledge. The proposed method mitigates these problems by implicitly learning transition dynamics (without explicitly modeling future trajectories) from offline data in a self-supervised manner based on the prediction of random features.", "questions": "1. I am confused on your statement that the Q-function for multi-step returns is independent of the policy when H = \\tauThe lim. You still have \\tilde{a}_1, \\tilde{a}_2, ..., \\tilde{a}_\\tau. From what policy do these actions come from?\n\n2. Why do you not compare against meta-RL methods? The successor features approach you compare against may count as meta-RL (it is not called that in their original paper though), but it is a method from 2017.", "limitations": "The limitations have been addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The methodology is interesting. Using random features in a self-supervised manner to form a Q-basis such that one can find a linear combination of these random features for the reward function of a downstream task using approximate multi-step returns makes sense and seems novel.", "weaknesses": "The main weakness of this approach lies in the experiments and poor framing of how this work fits in the context of meta-RL. meta-RL is not mentioned, but it entails this same problem of having multiple tasks with shared dynamics such that the reward function changes from one task to another. As such, the authors should be comparing against meta-RL benchmarks, not standard RL. It is interesting because the authors are running their experiments on a popular meta-RL benchmark called Meta-World and yet they do not compare their approach against meta-RL approaches. One of the reasons why this is currently an unfair comparison is the need for the competing approaches to learn their value functions, etc. from scratch, whereas the proposed approach had the benefit of learning from an offline dataset befrorehand and adapting to the target task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We respond to your comments below. \n\n> The main weakness of this approach lies in the experiments and poor framing of how this work fits in the context of meta-RL. meta-RL is not mentioned, … beforehand and adapting to the target task.\n\nThanks for the suggestion. We did not compare with meta-RL as usually *rewards* are needed for this case (in both the meta-training and meta-testing phases), while using a self-supervised approach without requiring reward labels is exactly one of the advantages of our approaches. By not requiring reward, our method avoids assumptions about train/test time reward sharing the same distribution, as is typical in meta-RL. Given this problem setting of having unlabelled offline datasets and distribution shift between meta-train and meta-test, comparing these settings is not quite apples to apples. However, as per your suggestion, we have added new experimental results and comparison with meta-RL algorithm RL2 in figure 1 of rebuttal pdf. We note that since our problem setting isn't typical meta-rl, which assumes the train/test time objectives are in the same underlying distribution, we have to adopt RL2 by giving it privileged information similar to our adoption of CQL. Out of the four environments which we ablate RL2, it's similar to our method in door open but inferior in all others. This shows that meta-rl will suffer from out-of-distribution goals which our method avoids.  \n\n> I am confused on your statement that the Q-function for multi-step returns is independent of the policy when $H = \\tau$ The lim. You still have $\\tilde{a}_1, \\tilde{a}_2, ..., \\tilde{a}_\\tau$. From what policy do these actions come from?\n\nGreat question! In a typical Q function $Q^\\pi(s, a)$ refers to expected value when a is taken at s, and the policy \\pi is followed thereafter. This makes the dependence on the policy $\\pi$ implicit, and the same Q-function cannot be used to evaluate a different policy. In a multi-step open-loop Q-function $Q(s, \\tilde{a}_1, \\tilde{a}_2, ..., \\tilde_{a})$, the dependence on *actions* is *explicit*, i.e., the Q-value, is completely *determined* directly by this sequence of actions (and does not have any implicit dependence on any specific policy $\\pi$). When trying to find the optimal sequence of actions, different sequences of actions are generated randomly and then evaluated using the learned Q-function to choose the *best* sequence of actions. Essentially, any policy that can generate the sequences of actions with a good coverage of all possible sequences would be sufficient. In doing so, the same multi-action Q function can evaluate many different action sequences within the data, without having any implicit policy dependence like standard Q-learning. An implicit policy dependence would make it hard to use the same Q function to evaluate many different action sequences. \n\nPlease feel free to let us know if there are any other comments that may help reevaluate our paper.\n"}]

Author rebuttal:
We thank the reviewers for their comments and suggestions. We highlight our main experimental additions here and then address individual reviewer concerns in each reviewer response: 

- **Meta-RL baseline (Reviewer 8qBK)**: We conduct a comparison with a meta-RL baseline, RL2 [1] that performs recurrent meta-learning. We train RL2 on a set of training tasks and evaluate its adaptation performance on out-of-distribution tasks to demonstrate the transfer behavior under distribution shift. Since RL2 adapts in a very short context (2 episodes), we plot the results at the end of adaptation as horizontal lines with error bars. From Fig. 1, we see that even with privileged information during training, RL2 does not handle distribution shift during testing and often performs poorly on test-time tasks. We note that the good performance on Door Open results from the goals being largely in the same direction. So even with the task separation, RL2 still bootstraps behavior from the training tasks. 
- **MOPO baseline (Reviewer BJ94)**: We conduct a comparison with an offline model-based RL baseline as requested, MOPO [2]. While this is not quite in the same problem setting, we adapted using the same assumptions as in CQL, running a goal conditioned variant of this method. As shown in Fig 1 in the rebuttal PDF, we found that RaMP outperforms MOPO across all Metaworld tasks. 
- **IQL baseline (Reviewer gqpD)**: We ran an IQL baseline on Metaworld domains as shown in Fig 1. We found that RaMP significantly outperformed IQL in both efficiency and asymptotic performance. 
- **D4RL experiments (Reviewer gqpD)**: We conducted experiments using the D4RL dataset on 2 maze environments - U Maze and Medium Maze name them. It is important to note that we are not in the standard offline RL setting, since we do not assume known reward on the offline dataset, only some at adaptation time. Moreover, the focus of RaMP is really on the transfer performance, so the standard D4RL comparisons are not quite comparing apples to apples. Our results show that RaMP is applicable to standard offline datasets in D4RL. 
- **Ablations wrt added implementation details (Reviewer BJ94)**: We conducted an ablation, removing each of the added components in the implementation details outlined in Section B. We find that each of the design contributes to the overall performance of our method. 

We provide additional clarifications, explanations and discussion in the per-reviewer responses. 

[1] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, Pieter Abbeel. RL2: Fast Reinforcement Learning via Slow Reinforcement Learning. ICLR 2017.

[2] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, Tengyu Ma. MOPO: Model-based Offline Policy Optimization. NeurIPS 2020.


Meta-review: There were some mixed opinions on the papers, but overall the positive aspects outweigh the negative. One point of difficulty in assessing the paper was that the problem setting did not allow for easy direct comparisons to algorithms that superficially appear to be comparable. The authors did a reasonable job in articulating this in the rebuttal and with follow-up experiments. They should take this point of confusion into account, as well as other critical points made by reviewers, when preparing the camera ready paper.
Decision: Accept (poster)