Model-free Reinforcement Learning that Transfers Using Random Reward Features
Authors: Boyuan Chen
Chuning Zhu
Pulkit Agrawal
Kaiqing Zhang
Abhishek Gupta
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement Learning
Model-free
Transfer
Random Features

Abstract: Favorable reinforcement learning (RL) algorithms should not only be able to synthesize controller for complex tasks, but also transfer across various such tasks. Classical model-free RL algorithms like Q-learning can be made stable, and has the potential to solve complicated tasks individually. However, rewards are key supervision signals in model-free approaches, making it challenging in general to transfer across multiple tasks with different reward functions. On the other hand, model-based RL algorithms, naturally transfers to various reward functions if the transition dynamics are learned well. Unfortunately, model-learning usually suffers from high dimensional observations and/or long horizons due to the challenges of compounding error. In this work, we propose a new way to transfer behaviors across problems with different reward functions that enjoy the best of both worlds. Specifically, we develop a model-free approach that implicitly learns the model without constructing the transition dynamics. This is achieved by using random features to generate reward functions in training, and incorporating model predictive control with open-loop policies in online planning. We show that the approach enables fast adaptation to problems with completely new reward functions, while scaling to high dimensional observations and long horizons. Moreover, our method can easily be trained on large offline datasets, and be quickly deployed on new tasks with good performance, making it more widely applicable than typical model-free and model-based RL methods. We evaluate the superior performance of our algorithm in a variety of RL and robotics domains.

Under review as a conference paper at ICLR 2023
MODEL-FREE REINFORCEMENT LEARNING THAT TRANSFERS USING RANDOM FEATURES
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning (RL) algorithms have the potential not only for synthesiz- ing complex control behaviors, but also for transfer across tasks. Typical model- free RL algorithms are usually good at solving individual problems with high di- mensional state-spaces or long horizons, but can struggle to transfer across tasks with different reward functions. Model-based RL algorithms, on the other hand, naturally enable transfer across different reward functions, but struggle to scale settings with long horizons and/or high dimensional observations. In this work, we propose
way to transfer behaviors across tasks with different reward functions, displaying the benefits of model-free RL algorithms with the transfer- ability of model-based RI.. In partieular, we show how careful combination of model-free RL using randomly sampled features as reward is able to implicitly model long-horizon environment dynamics. Model-predictive control using these implicit models enables quick adaptation to problems with new reward functions, while scaling to problems with high dimensional observations and long horizons. Our method can be trained on offline datasets without reward labels, and quickly deployed on new tasks, making more widely applicable than typical methods for both model-free and model-based RL. We validate that our proposed algo- rithm enables transfer across tasks in a variety of robotics and analytic domains.
INTRODUCTION
Reinforcement learning (RI.) algorithms have been shown to successfully synthesize complex be- havior in single-task sequential decision-making problems 1
3 more importantly have the potential for broad generalization across problems. However, many RL algorithms are deployed as specialists - they solve single tasks and are not prepared for reusing their interactions. this work, specifically focus the problem of transferring information across problems where the environ- ment dynamics are shared, but the reward function is changing. This problem setting is reflective of number of scenarios that may be encountered in weal-world settings such robotics. For instance, in tabletop robotic manipulation, different tasks like pulling an object, pushing an object, picking it up, and pushing different locations, all share the same transition dynamics, but involve changing reward function. We hence ask the question can we reuse information across these tasks in way that scales to high dimensional, longer horizon problems?
When considering how to tackle this problem, natural possibility is to consider direct policy search 5). Typical policy search algorithms can achieve good performance for solving single task, but entangle the dynamics and reward, in the sense that the policy one searches for is optimal for particular reward but may be highly suboptimal in new scenarios. Other model-free RL algorithms like actor-critic methods 6 g Q-learning [9] I may exacerbate this issue, with learned Q-functions entangling dynamics, rewards, and policies. For new scenarios, an ideal algorithm should be able to disentangle and retain the elements of shared dynamics, while being able to easily substitute in new rewards.
A natural fit disentangle dynamics and rewards are model-based RL algorithms 100
13
14]. These algorithms usually learn a single-step model of transition dynamics and leverage this leamned model to perform planning [15 12 II, [6]. These models are naturally modular and can be used to re-plan behaviors for new rewards. However, one-step dynamics models are brittle and suffer from challenges in compounding error [8]

---

Under review as a conference paper at ICLR 2023
In this work, we ask - can we build reinforcement learning algorithms that disentangle dynamics, rewards, and policies for transfer across problems but retain the ability to solve problems with high dimensional observations and long horizons? In particular, we propose algorithm that can train on large offline datasets of transitions in an environment at training time to implicit model transition dynamics, and then quickly perform decision making on variety of different new tasks with varying reward functions that may be encountered at test time.
Specifically, we propose model the long-term behavior of randomly chosen basis functions (often called cumulants) of the environment state and action, under open-loop control, using what we term Q-basis functions. These Q-basis functions can be easily recombined to infer the true Q function for tasks with arbitrary rewards by simply solving linear regression problem. Intuitively, this suggests that rather than predicting the evolution of the entire state step by step, predicting the accumulated long-term future of many random features of the state contains information equivalent to dynamics model, thereby forming an "implicit model" that can transfer. These implicit models scale better with horizon and environment dimensionality than typical one-step dynamics models, while retaining the benefits of transferability and modularity.
Our proposed algorithm Random Features for Model-Free Planning (RaMP) allows us to leverage an unlabelled offline dataset to learn reward-agnostie implicit models that can quickly solve new tasks involving different reward functions in the same shared environment dynamics. We show the efficacy of this method on a number of tasks for robotic manipulation and locomotion in simulation, and highlight how RaMP provides more general paradigm than typical generalizations of model- based or model-free reinforcement learning.
1.1 RELATED WORK
Model-based RL is naturally suited for this transfer learning setting, by explicitly learning a model of the transition dynamics and the reward function [2
21. These models are typically learned via supervised learning on one-step transitions and are then used to extract control actions via planning 222 23 or trajectory optimization [24 25| 26). The key challenge in scaling lies in the fact that they sequentially feed model predictions back into the model for sampling 18 17. This can often lead to compounding errors [7 28), which grows with the horizon length unfavorably. In contrast, our work does not require autoregressive sampling, but directly models long term behavior, and is easier to scale to longer horizons and higher dimensions.
19 16
20
On the other hand, model-free RL often avoids the challenge of compounding error by directly mod- eling either policies or Q-values [4] 29 30 7 and more easily scales to higher dimensional state spaces 53] However, this entangles rewards, dynamics, and policies, making challenging to directly use for transfer. While certain attempts have been made at building model-free methods that generalize across rewards, such as goal-conditioned value functions (32 33
34 35] or multi-task
policies B3 37], they only apply to restricted classes of reward functions and particular training distributions. Our work aims to obtain the best of both worlds (model- and model-free RL), learning disentangled representation of dynamics that is independent of rewards and policies, but using a model-free algorithm for learning.
Our notion of long-term dynamics is connected to the notion of state-action occupancy measure [38 39.. often used for off- off-policy evaluation and importance sampling methods in RL. These methods often try to directly estimate either densities or density ratios [14 38 39] Our work simply learns the long-term accumulation random features, without requiring any notion of normalized densities.
Perhaps most closely related work to ours is the framework of successor features, that considers transfer from a fixed set of source tasks to new target tasks [40] 41
43). Like our work, the successor features framework leverages linearity of rewards disentangle long-ten term dynamies from rewards using model-free RI.. However, transfer using successor features is critically dependent on choosing (or learning) the right featurization and entangles the policy. Our work leverages random features and open-loop policies to allow for transfer across arbitrary policies and rewards.
BACKGROUND AND SETUP
Formalism: We consider the standard Markov decision process (MDP) as characterized by a tuple M = (S,A,7 T, R, v, ni), with state space S, action space A, transition dynamics T : S x A
2

---

Under review as a conference paper at ICLR 2023
(SS), reward function R S x A A(l-Rmax: Rmax)), discount factor y [0, 1), and initial state distribution μl E A(S). The goal is to learn a policy т : :S - A(A), such that it maximizes the expected discounted accumulated rewards, i.e., solves max; 1 te > with rh == ((5%, 04) Ray,as Pr(-I 8h. o²). Hereafter, we will refer an MDP and a task interchangeably.
Estimating Q-functions: Given an MDP M. one can define the state-action Q-value function under any policy π as Q=(s,a) == ap-a(-19s) (sn)
Eh= r 8i = s,a, = a which
#641 (-.()))).n)))
denotes the expected accumulated reward under policy т, when starting from state-action pair (s,a). Similarly, one can also define the multi-step (T-step) Q-function Qr 02, ar)
E Par+aro(-1ssrd eh-InT(- (iiaas)
a1
= 02 022 a4 0T
One can estimate the Q+ by Monte-Carlo sampling of the trajectories under x, i.e. by solving
1 E|@000 at)
1
8
m 2
min QEQ
(2.1)
N
M
where is some function class for Q-value estimation, which in practice is some parametric func- tion class, e.g., neural networks; r% R.mi ams and (5), (si .% m.j) come from MN trajectories that are generated by N open-loop action sequences {(ā). 2z ................................
For each sequence there are M trajectories starting from it, and following policy onwards, to estimate the 7-step Q-function. A large body of work considers finding this Q-function using dynamic programming, but for the sake of simplicity, this work will only consider Monte-Carlo estimation.
In practice the infinite-horizon estimator in 2 can be hard to obtain. We hence use a finite- horizon approximation of Q. (of length H), denoted by Q in learning. Note that if one chooses H = T, then the T-Step Q-function defined above becomes Q (s, "2, ; ==
E sn))a))
H! Σ#
Th
81 = s,01=
ag = äæ Note that in this case, the Q-
function is irrelevant the policy क, denoted by Q" and is just the expected accumulated reward under the open-loop action sequence (ā1, ...... ... aæ ). This Q-function can be used to score how "good" a sequence of actions will be, which in turn can be used for planning.
2.1 PROBLEM SETUP
We consider transfer and offline RL scenario, where assume access an offline dataset consist- ing of several episodes D = {( 4% sm ( This dataset assumes that all transitions collected under the same transition dynamics T, but otherwise does not require labels for re- wards, and may come from multiple different behavior policies as well. Here H is the length of the trajectories, which is large enough, c.g., of order ((1((1 - 7)) approximate the infinite-horizon setting; M is the total number of trajectories.
The goal is to make the best use of the dataset D, and generalize the learned experience im- prove the performance on a new task M, with the same transition dynamics T but arbitrary reward functions R. Note that unlike some related work 40 44 we make no assumption on the reward functions of the MDPs that generate D, i.e., these MDPs do not have to share any structure of the reward functions, e.g., being linear in some common features.
fact, the samples of the rewards that correspond to the trajectories in D are not even necessary. The goal of the learning problem is to pre-train on the offline dataset such that we can enable very quick (even zero-shot) adaptation to the new reward functions encountered at test time.
3 RAMP: LEARNING IMPLICIT MODELS FOR CROSS-REWARD TRANSFER WITH MODEL-FREE TECHNIQUES
In this section, we introduce algorithm, Randomized features for Model-free Planning (RaMP), to solve the problem described in 2 learning a model of long-term dynamics that enables transfer to tasks labeled with arbitrary new rewards, while mitigating challenges with compounding error.
3

---

Under review as a conference paper at ICLR 2023
Offline Training
Online Adaptation to Any New Objective
== (15 " dr)
D
Exploration Trajactories without Reward
(s,, a,) (g2.ag)
18,, Ry)
(5 dr,
Small Amount of Interaction
Linear Regression
Random Feature
300
(w. (w.
Ve
y aggregated feature
Model Predictive Control
Infer Qrest (5.0,,.. aq.
Figure I: RaMP: Depiction our proposed method for transferring behavior across tasks by leveraging model- free learning of random features. At training time, Q-hasis functions are trained on accumulated random fea- tures. At test time, adaptation is performed by solving linear regression and recombining basis functions, followed by online planning with MPC.
We start by arguing where model-based and model-free algorithms fall short. Model-based RI. approaches estimate the transition dynamics T using the data in D, and plan in the estimated model. The key advantage of this approach is that it is reward-agnostic, and has the potential to easily generalize to multiple tasks. Unfortunately, since the model outputs are fed back into the model for multi-step planning, it is subject to compounding error of one-step dynamics models [17].
In contrast, one can resort to model-free RL approaches, c.g., Q-learing or policy optimization methods 66. 9 4, directly optimize the value of interest. These methods less subject to the challenges of compounding error than most model-based ones. Empirically, learning neural networks predict the Q-function (a scalar for each 5,,)), can be much easier than predict the next state (which can be a high-dimensional vector, e.g., image). However, these methods cannot be used directly transfer across different tasks with different rewards, as they are designed to be reward- dependent. This raises the natural question: Is there a model-fre approach that can mitigate the challenges of compounding error and can transfer across tasks painlessly?
The key insight we advocate is that if instead of modeling long term accumulation of some specific reward as a Q-function, we directly model long-term accumulation of many random features of state-actions under arbitrary open-loop action sequences. This can effectively disentangle transition dynamics, reward, and policies being evaluated, and potentially allow for transfer across tasks. Each long-term accumulation of random features is referred as element of a "random" Q-basis, and can be learned with simple modifications to typical model-fres RL algorithms.
At training time, the offline dataset D can be used to learn set of "random" Q-basis functions for different random features. This effectively forms an "implicit model", as carries information about how the dynamics propagates, without being tied any particular reward function policy. At test time, given a new reward function, we can recombine Q-basis functions linearly to effectively approximate the true reward-specific Q-function. This inferred Q-function can then be used for planning for the new task.
3.1 OFFLINE TRAINING: LEARNING RANDOM Q-FUNCTIONS FROM UNLABELLED DATA
Given a dataset of transitions without reward labels, the goal of this phase is to model the long-term accumulation of random features under random state-action sequences. With prior knowledge about the downstream test-time rewards, the random features being modeled must be expressive and universal in their coverage, that any possible test-time rewards can be reconstructed from these random features by linear regression. As suggested in [45 46 47 random features can be powerful in representing nonlinear functions, i.e., any test-time reward function in our case, their linear combinations. In particular, suppose we have K neural networks o0.; 0z) S x A - IR with weights 0z E R and k [K]. where 0g randomly i.i.d. sampled from some distribution p. Sampling K such weights 0₂ with k e [K] yields vector of scalar functions (e(s,:; 0x 0x)]kE[k] E RR for any (s, a). which can be used as random features whose accumulation through dynamics can be used to model Q-bssis functions.
To model the long-term accumulation of each of these random features, we note that they can be treated reward functions in model-free RI,, and the machinery of Q-functions can be reused to
4

---

Under review as a conference paper at ICLR 2023
learn their long-term accumulation. As discussed in 48 model-free RL algorithms can be used to model the evolution of arbitrary functions (called "cumulants") >) the state. Therefore, we can learn set of K Q-basis functions, with each of them corresponding to particular random feature.
We note that this definition of a Q-basis function tied to a particular policy π that generates the trajectory. To transfer, of actions, as the optimal policy for the new task is likely to not be within the span of policies seen in training. To allow the modeling of cumulants that independent of particular policies, propose to learn open-loop Q-basis functions for each of the random features (as discussed in Section $2). which is policy-agnostic, and can be used to search for optimal actions new tasks.
needs to predict the accumulated random features under sequences
To actually learn these Q-basis functions (one for each random feature), we opt to use Monte-Carlo methods for simplicity. We generate a new dataset Do from D, with Do
{((T]) ; aT.n), ΣnE(h) 105 aa;;; ( me)))mm(MM)Ae)) Enem MM-16(sT, ax;;; 0k) as the accumulated cumulants for open-loop action sequences {ai.... ..1) taken from state $1- We then use K function approximators representing each of the K Q-basis functions, e.g. neural networks (b(,,;; vx)) S x A - R for k E [K], to fit the accumulated cumulants. Specifically, we minimize the following loss
Here use
min (Pc
Σ
(bO0 , (THiP)
Σ RE(H]
a ~
ai :.:0))))
(3.1)
M
ME(AA]AEE(]
These Q-basis functions can recombined to approximate the Q-functions for true rewards at test time. The two key design decisions here are (1) predicting the evolution of random features, rather than one-step modeling of state, and (2) predicting the accumulated random features under open-loop action sequences, rather than a closed-loop policy.
3.2 ONLINE PLANNING: INFERRING Q-FUNCTIONS WITH LINEAR REGRESSION AND PLANNING WITH
CONTROL
The goal of learned Q-basis functions is enable transfer new tasks with arbitrary rewards. Any reward function can be approximately expressed as a linear combination of a sufficiently ex- pressive and expansive set of random features. Given this linear approximation, we can recover an approximation to the Q-function for the true test-time reward by recombining the random Q-basis functions linearly. Therefore, we can obtain the test-time Q-function by solving simple linear regression problem. This inferred Q-function can then be used to obtain an optimal sequence of actions through planning.
3.2.1 REWARD FUNCTION FITTING WITH RANDOMIZED FEATURES
We first learn how express the reward function for the new task as linear combination of the ran- dom features. This can be done by solving a linear regression problem to find the coefficient vector [w,... ... wx] that approximates the new task's 's reward function linear combination of the random features. Specifically, we minimize the following loss
Σ
(r(o .
Σ RE(K]
wi unddss,., ,N: + A|[w|| (3.2)
w = argmin
MH
ME(M)
where À > is the regularization coefficient, and r(s ox Roy.em. Due to the use of random features, Eq. (3.2) is ridge regression problem, and can be solved efficiently. In case the reward labels are being obtained on the fly during online data collection, we can leverage an online least squares algorithm [49 to continually improve our estimate of w without re-computing regression result from scratch as the number of samples grows.
Given these weights, it is easy to estimate an approximate open-loop Q-function for the true re- ward on the new task by linearly combining the Q-basis functions learned in the offline train- ing phase {b((, v/) ( kejKK] according to the same coefficient vector w*. This follows from the additive nature of reward and linearity of expectation. In particular, if the reward function a) ΣtE(K] wod(s, a; 0k) holds approximately, which will be the case for large enough K and rich enough ø, then the approximate Q-function for the test-time reward under
5

---

Under review as a conference paper at ICLR 2023
the sequence {ar.-.. ,n) satisfies Q" (51 (1:H) == Ea Ene(n 7h Ray
EnE[K] wa (s1, a1:H; : where {wi ]e[[ is the solution the regression problem (Eqn (32))
and {vi [E(KK] is the solution to the Q-basis fitting problem (Eqn (3.1)
3.2.2 PLANNING WITH PODEL-PREDDITIVV CONTROL
To obtain the optimal sequence of actions we can use the inferred approximate Q-function for the true reward Q" (s1, (1:H) for online planning each time the new task: at state 8;, we conduct standard model-predieee control with random shooting, i.e., randomly generating N sequences of actions {a", 2/ InE[N), and find the action sequence with the maximum Q-value such that
ni E &argmax nE(N]
Σ KE
w (8t, o G
(3.3)
" from the sequence ni, observe the new state S1+1e and replan. Our algorithm We refer readers to Appendix D for a detailed connection of our proposed method to existing work and Appendix A for detailed pseudocode.
We then execute a, summarized Algorithm [
3.3 THEORETICAL JUSTIFICATIONS
We now provide some theoretical justifications for the methodology adopt. To avoid unnecessary nomenclature of measures and norms in infinite dimensions, we in this section consider the case that and A are discrete (but be enormously large). Due space limitation, we present abridged version of the results below, and defer the detailed versions and proofs in SC We first state the following result on the expressiveness of random cumulants.
Theorem 3.1 (Q-function approximation; Informal), Under standard coverage and sampling as- sumptions of offline dataset D, and standard assumptions the boundedness and continuity of random features d(s, a; 0), follows that with horizon length H Õ log( and M
1-7
Õ( ((-))))))) episodes dataset D, and with K = 5((1 - -2 22 random features, we have that for any given reward function R, and any policy π
11"" (w²) Qx I 50(e) O
inf E(ʃ) EEH
with high probability, where for each (s,a), 2 (s, a;w") is defined @ (s, a; wqw"" E Em 2-1 EKE(K) w ani 02) s, a and can be estimated from the offline
dataset D; yen E(f) is the infimum expected risk over the function class H/ induced by d.
Theorem 3.1 is an informal statement of the results in which specifies the number of random features, the horizon length per episode, and the number of episodes, order to approximate Q, accurately by using the data in D, under any given reward function R and policy Note that the number random features is not excessive and polynomial in problem parameters. We also note that the results be improved under stronger assumptions of the sampling distributions p and kernel function classes [47 50].
Next, we justify the use of open-loop Q-functions in planning in deterministic transition environ- ment, which contains all the environments our empirical results we will evaluate later. Recall that for any given reward R. let Q, denote the Q-function under policy æ. Note that with slight abuse of notation, Qn can also be the multi-step step Q Q-function (see definition in $2). and the meaning should be clear from the input, i.e., whether it is (s, a) (s, ai, ,aH).
Theorem 3.2. Let be some subclass of Markov stationary policies, i.e., for any π E II, ग S ((A). Suppose the transition dynamics T is deterministic. For any given reward R, denote the H- step policy obtained from H-step open-loop policy improvement over as Th S + A defined T (s) argmax an an) E max Q= "1,... ,an), for all S. Let Va, denote the

---

Under review as a conference paper at ICLR 2023
We evaluate method on manipulation, locomotion, high dimensional action space environments. The green arrow in each environment indicates the online objective for policy transfer while the red arrows offline objectives used to label rewards for the privileged dataset.
Figure 2:
value function under the open-loop policy THH (see formal definition in C.11 Then, we have that for all S
Vain (s) max max Q= 01,' .aH) max max (s, R1: HEll
HEll
The proof of Theorem 3.2 can be found in The result can be viewed as generalization of the generalized policy iteration result 40 to multi-step open-loop policies. Taking II be the set of policies that generate the data, the result shows that the value function of the greedy open-loop policy improves over all the possible H-step open-loop policies, with the policy after step to be any policy in II. Moreover, the value function by T'r also improves overall one-step policies if the policy after the first step onwards follows any policy in II. This due to the fact that (coming from data) might be much more restricted policy class than any open-loop sequence 01:H.
4 EXPERIMENTAL EVALUATION
In this section, aim to answer the following research questions: (1) Does RaMP allow for effective transfer behaviors across tasks with varying rewards but shared dynamics?, (2) Does RaMP seale to domains with high dimensional observation spaces and longer horizons?, (3) Does RaMP scale to domains with high dimensional action space? (4) Which design decisions in RaMP enable better transfer and scaling?
4.1 EXPERIMENT SETUP
Across several domains, we evaluate the ability of RaMP to leverage the knowledge of shared dy- namics from an offline dataset to quickly solve new tasks with arbitrary rewards.
Offline Dataset Construction: For each domain, we have an offline dataset collected by a behavior policy as described Appendix B.2 Typically this behavior policy a mixture of noisy policies accomplishing different objectives each domain. Although RaMP and other model-based methods do not require knowledge of any reward from the offline dataset and simply require transitions, other baseline comparisons will require privileged information. Baseline comparison methods like model- free RL and successor features require the provision of a set of training objectives, as well as rewards labeled for these objectives state-actions from the offline dataset. We call such objectives 'offline objectives' and dataset annotated with these offline objectives and rewards a privileged dataset.
Test-time adaptation: At test-time, we select novel reward function for online adaptation, referred to as 'online objective' below. The online objective may correspond to rewards conditioned on different goals or even arbitrary rewards, depending on the domain. Importantly, the online objective need not be drawn from the same distribution as the privileged offline objectives above.
Given this problem setup, we compare RaMP with a variety of baselines. (1) MBPO [13] is a model- based reinforcement learning method that learns standard one-step dynamics model and uses actor- critic methods to plan the models. We pre-train the dynamics model for MBPO on the offline dataset before running the full algorithm on the testing environment, (2) Successor feature (SF) [40) framework for transfer learning in RL as described in Sec. I.| SF typically assumes access to set of policies towards different goals along with a learned featurization, the privileged dataset to learn a set of policies corresponding to the offline objectives using offline reinforcement learning [51]. We also leam successor features with the privileged dataset [40]. (3) CQL [5T]: As an oracle comparison, we compare with a coal-conditioned variant of an offline RL algorithm (CQL.). CQL is model-free offline RI algorithm that learns policies from offline data.
provide it with

---

Under review as a conference paper at ICLR 2023
1o2
Reach Wall
1o2
Door Open
162
Faucet Open
1c2
Button Press
Return
1.5
1.0
0.5
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e3
Hopper Sprint
le4
1o3
Hopper Jump
1e4
lesHopper Backward 2ee
1o2
D'Claw
1e4
Episodo Return
4-
2
-1
-2
1.0 Environment Steps 1e4
0.5
0.5
1.0
0.5
1.0
0.0
0.5
1.0
Environment Steps 1e4
Environment Steps 1c4
Environment Steps 1e4
RaMP (Ours)
MBPO
SF
CQL
Figure 3: Reward transfer results on Metaworld, Hopper and D'Claw environments. RaMP adapts to novel rewards more rapidly than MBPO, Successor Features, and CQL baselines. More experiments are in appendix.
While model-free offline RL naturally struggles to adapt to arbitrarily changing rewards, we instead afford CQL additional privileges by providing it with information about the goal at both training and testing time. CQL is then trained on the distribution of training goals on the offline dataset, and finetuned on the new goal provided at test time. In this sense, the CQL comparison is assuming access more information than RaMP, Each method is benchmarked on each domain with 9 seeds.
4.2 TRANSFER TO NOVEL REWARDS
We first evaluate the ability of RaMP to learn from an offline dataset and quickly adapt to novel test rewards in 4 robotic manipulation environments from meta-world [52]. We consider skills like reaching target across the wall, opening door, turning a faucet, and pressing button while avoiding obstacles, which are challenging for typical model-based RL algorithms (Fig. (
Each domain features 50 different possible goal configurations, each associated with different reward but the same dynamics. The privileged offline objectives consist of 25 goal configurations described in 1 The test-time reward functions drawn from the remaining 25 "out-of- distribution" reward functions. We refer the reader to Appendix B. for details of this setup.
As shown in Fig our method adapts to reward most quickly across all four domains. MBPO slowly catches up with our performance with more samples, since it still needs to learn the Q function from scratch even with the dynamics branch trained. In multiple environments, successor features barely transfer to the online objective as it entangles policies that aren't 't close to those needed for the online objective. Goal-conditioned CQL performs poorly in all tasks as faces a hard time generalizing to out-of- of-distribution goals. In comparison, RaMP is able to deal with arbitrary sets of test time rewards, since it does not depend on the reward distribution at training time.
4.3 SCALING TO TASKS WITH LONGER HORIZONS
We further evaluate the ability of our method scale to tasks with longer horizons. We consider locomotion domains such as the Hopper environment from OpenAl Gym [53]. We chose the of- fline objectives be running forward at different velocities. The online objectives for adaptation correspond to novel skills such as standing, sprinting, jumping, running backward. Among them, standing and sprinting are goal-conditioned objectives that correspond to running forward zero and maximum speed, while jumping and running backward have drastically different objectives that are difficult to express as parametric "goals". Therefore, goal-conditioned methods like CQL are not applicable jumping and running backward. As shown in Fig. 3 our method maintains the highest performance when adapting to drastically different online objectives, as it is designed to make assumption about reward in the offline dataset, while avoiding compounding errors by di- rectly modeling accumulated random features. MBPO fails to match the performance of RaMP since higher dimensional observation and longer horizon increase the compounding error of model-based
8

---

Under review as a conference paper at ICLR 2023
methods. We note that SF is performing reasonably well, likely because the method also reduces the compounding error compared to MBPO. Furthermore, its featurization trained with privileged data and thus still captures useful information for the online objectives. In Appendix [.4 we fur- ther test our method on environments with even higher dimensional observations, such as image observations. We refer the reader to Appendix B.1 for further details.
4.4 SCALING TO HIGH DIMENSIONAL STATE-ACTION SPACES
To understand whether RaMP can scale to higher dimensional state-action spaces, we consider a dexterous manipulation domain (referred to the D' Claw domain in Fig This domain has a DoF action space controlling each of the joints the hand as well as a 16-dimensional state space including object position. The offline dataset is collected moving the object to different orientations, and the test-time rewards are tasked with moving the object to new orientations (as described in Appendix B.I). Fig 3 shows that both Q-estimation and planning with model-predictive control remain effective when action space is large.
4.5 ABLATION OF DESIGN CHOICES
To understand what design decisions in RaMP enable better transfer and scaling, we conduct ablation studies on various domains, including an analytical 2D point goal-reaching environment and its variants (described in Appendix [B.I). as well as the classic Inverte@Doubleeenduul domain and meta-world reaching. We report an extensive set of ablations in Appendix B.5
Point
Reach
Hopper
Pendulum
Random
34.0 + 1.0
820.8 142.0
Ours
25.7 + 5.4
65.1 = 0.4
Gaussian
-3.6 + 7.4
188.6 + 49.2
MBRL
50.2 + 9.7
395.4 + 5.8
Polynomial
24.5 + 5.8
162.9 36.4
Table I: Policy ovaluation error. Feedforward dy- namics model suffers from compounding error that is particularly noticeable in domains with high ac- tion dimensions chaotic dynamics. Our method achieves low approximation error in both domains.
Table 2: Return for different features. Random fea- tures are able approximate the true reward well across domains. Polynomial features work in simple environments but do not scale to complex rewards. Gaussian features unable to express the reward.
Reduction of compounding error with open-loop Q functions We hypothesize that our method does not suffer from compounding errors in the same way that feedforward dynamics models do. In Table we compare the approximation error of truncated Q values computed with (1) open-loop Q functions obtained as a linear combination of random cumulants (Ours), and (2) rollouts of feedforward dynamics model (MBRL). We train the methods on offline data and evaluate on data from novel task at test time. Note that this setting is analogous to performing policy evaluation the behavioral policy induced by an offline datasct. We perform the comparison on one environment with high action dimension (Hopper) and one with chaotic dynamics (Pendulum). As shown in Table our method outperforms feedforward dynamics models.
Effect of different types of featurization We experiment with three choices of projections: random features parametrized by deep neural network, random features parametrized by a gaussian matrix, and polynomial features of state and action up the second order. We evaluate these choices on Point and Metaworld Reach in Table 2 We see that NN-paramerrized random features approximate the true reward well as a linear combination of random features for all three tasks. Polynomial features perform well on environments with simple rewards that are linear in polynomial basis, but struggle as the reward becomes more complex, while Gaussian features are rarely expressive enough.
DISCUSSION
In this work, we introduce RaMP, method for leveraging diverse prior offline data to learn models of long horizon dynamics behavior, while being able to naturally transfer across tasks with different reward functions. To do so, we combine the best elements of model-based and model-free reinforce- ment learning. By learning the long-term evolution of random features under open loop policies, we able to disentangle dynamics, rewards, and policies. We show how this technique allows us to learn behavior that naturally transfers across tasks, even under misspecification of reward func- tions. Across a number of simulated roboties and control domains, RaMP achieves superior transfer

---

Under review as a conference paper at ICLR 2023
ability than baseline comparisons. In future work, hope explore how to combine RaMP with more powerful planning methods like [54] and dynamic program techniques for learning Q-basis functions [6]
REFERENCES
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
[2] Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: example- guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 37(4):143, 2018.
[3] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy P. Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. CoRR, abs/1712.01815, 2017.
[4] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Mach. Learn., 8:229-256, 1992.
[5] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.0034f) 2017.
[6] Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. In Sara A. Solla, Todd K. Leen, and Klaus-Robert Müller, editors, Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 December 1999], pages 1008 1014. The MIT Press, 1999.
[7] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290 2018.
[8] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error actor-critic methods. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference
Machine Learning, ICML 2018, Stockholmsmässnn Stockholm, Sweden, July 10-15, 2018, volume 80 Proceedings of Machine Learning Research, pages 1582-1591. PMLR, 2018.
[9] Christopher C. H. Watkins and Peter Dayan. Q-learning. In Machine Learning, pages 279- 292, 1992.
[10] Thomas M. Moerland, Joost Broekens, and Catholijn M. Jonker. Model-based reinforcement learning: A survey. CoRR, abs/2006. 16712, 2020.
[11] Mare Peter Deisenroth and Carl Edward Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In Lise Getoor and Tobias Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning, (CML2011 Bellevue, Washinglon, USA, June
28 - July 2, 2011, pages 465-472. Omnipress, 2011.
[12] Nicolas Heess, Gregory Wayne, David Silver, Timothy P. Lillicrap, Tom Erez, and Yuval Tassa. Leamning continuous control policies by stochastic value gradients. Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2944-2953 2015.
[13] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019. NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 12498- 12509, 2019.
[14] Michael Janner, Igor Mordatch, and Sergey Levine. ference learning for infinite-horizon prediction. In Hugo Larochelle, Marc' Aurelio Ranzato,
Generative temporal dif-
10

---

Under review as a conference paper at ICLR 2023
Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tier Lin, editors, Advances in Neural Infor- mation Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[15] Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, pages 7559-7566. IEEE, 2018.
[16] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. CoRR, 0llos/1805.122 2018.
[17] Nathan O. Lambert, Kristofer S. J. Pister, and Roberto Calandra. Investigating compounding prediction errors in learned dynamics models. CoRR, abs/2203.09633 2022.
[18] Kavosh Asadi. Dipendra Misra, and Michael L. Littman. Lipschitz continuity in model-based reinforcement learning. Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmüsson Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 264-273, PMLR, 2018.
[19] Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In 8th International Conference on Learning Rep- resentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[20] Richard S. Sutton. Integrated architectures for learning, planning, and reacting based on ap- proximating dynamic programming. In Bruce W. Porter and Raymond J. Mooney, editors, Machine Learning. Proceedings of the Seventh International Conference on Machine Learn- ing, Austin, Texas, USA, June 21-23, 1990, pages 216-224. Morgan Kaufmann, 1990.
[21] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1373 2016.
[22] Sergey Levine and Vladlen Koltun. Learning complex neural network policies with trajec- tory optimization. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 029-8377 JMLR.org. 2014.
[23] Oleh Rybkin, Chuning Zhu, Anusha Nagabandi, Kostas Daniilidis, Igor Mordatch, and Sergey Levine. Model-based reinforcement learning via latent-space collocation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9190-9201. PMLR, 2021.
[24] Hyung Ju Terry Suh, Tao Pang, and Russ Tedrake. Bundled gradients through contact via randomized smoothing. IEEE Robotics Autom. Lett., 7(2):4000-4007, 2022.
[25] Matthew Zucker, Nathan D. Ratliff, Anca D. Dragan, Mihail Pivtoraiko, Matthew Klingen- smith, Christopher M. Dellin, J. Andrew Bagnell, and Siddhartha S. Srinivasa. CHOMP: covariant hamiltonian optimization for motion planning. Int. J. Robotics Res., 3209-10):1164 1193, 2013.
[26] Michael Posa and Russ Tedrake. Direct trajectory optimization of rigid body dynamical sys- tems through contact. In Emilio Frazzoli, Tomás Lozano-Pérez. Nicholas Roy, and Daniela Rus, editors, Algorithmic Foundations of Robotics X Proceedings of the Tenth Workshop on the Algorithmic Foundations of Robotics, WAFR 2012, MIT; Cambridge, Massachusetts, USA, June 13-15 2012, volume 86 of Springer Tracts in Advanced Robotics, pages 527-542. Springer, 2012.
[27] Arun Venkatraman, Martial Hebert, and J. Andrew Bagnell. Improving multi-step prediction of learned time series models. In Blai Bonet and Sven Koenig, editors, Proceedings of the Twenty-Ninth AAAI Conference Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pages 3024-3030. AAAI Press, 2015.
[28] Chenjun Xiao, Yifan Wu, Chen Ma, Dale Schuurmans, and Martin Müller Learning to combat compounding-erron in model-based reinforcement learning. CoRR, abs/1912.1 11222.12000 11206, 2019.
11

---

Under review as a conference paper at ICLR 2023
[29]
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015.
[30] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy Lill- icrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria-Florina Florina Balcan and Kilian Q. Weinberger, editors, Proceed- ings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1928 1937. Jog, 2016.
[31] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continu- ous control: Improved data-augmented reinforcement learning. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25.29, 2022. Open- Review.net, 2022.
[32]
Leslie Pack Kaelbling. Learning to achieve goals. Citeseer.
[33] Marcin Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experi- ence replay. In Advances in Neural Information Processing Systems, pages 5048-5058, 2017.
[34] Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.03698, 2019.
[35] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria,
May 3-7, 2021. OpenReview.net, 2021.
[36] Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. CoRR, abs/2104.08212, 2021.
[37] Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-Firsi Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27- -Febuarry 2019, pages 3796- 3803. AAAI Press, 2019.
[38] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estima- tion of discounted stationary distribution corrections. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer. Florence d' A'AhhhéBBu Buché-Buc,,a Emily B. Fox, and Roman Garnett, editors, Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurIPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, 2019.
pages
[39] Ruiyi Zhang, Bo Dai, Lihong Li. and Dale Schuurmans. Gendice: Generalized offline estima- tion of stationary values. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[40] André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
[41] Lucas Lehnert and Michael L. Littman. Successor features combine elements of model-free and model-based reinforcement learning. Mach. Learn. Res., 21: 11:196: 2020.
[42] Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep reinforcement learning with successor features for navigation across similar environments. In 2017 IEEE/RSJ International Conference Intelligent Robots and Systems, IROS 2017, Van- couver, BC, Canada, September 24-28, 2017, pages 2371-2378. IEEE, 2017.
[43] Angelos Filos, Clare Lyle, Yarin Gal, Sergey Levine, Natasha Jaques, and Gregory Farquhar. Psiphi-learning Reinforcement learning with demonstrations using successor features and in- verse temporal difference learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual
12

---

Under review as a conference paper at ICLR 2023
Event, volume 139 of Proceedings of Machine Learning Research, pages 2021.
PMLR,
[44] Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning us- ing successor features and generalised policy improvement. In International Conference on Machine Learning, pages 501-510. PMLR, 2018.
[45] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances Neural Information Processing Systems, 20, 2007.
[46] Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing mini- mization with randomization in learning, Advances in Neural Information Processing Systems,
21, 2008.
[47] Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with random features. Advances in Neural Information Processing Systems, 30, 2017.
[48] Richard S. Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M. Pilarski, Adam White, and Doina Precup. Horde: scalable real-time architecture for Iearning knowledge from unsupervised sensorimotor interaction. In Liz Sonenberg, Peter Stone, Kagan Tumer, and Pinar Yolum, editors, 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2011), Taipei, Taiwan, May 2-6, 2011, Volume 1-3, pages 761-768. IFAAMAS, 2011.
[49] Todd K Moon and Wynn C Stirling. Mathematical methods and algorithms for signal process- ing. Number 621.39: 51 MON. 2000.
[50] Zhu Li, Jean-Francois Ton, Dino Oglic, and Dino Sejdinovic. Towards unified analysis of random Fourier features. In International Conference on Machine Learning, pages 3905-3914.
PMLR, 2019.
[51] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. CoRR, abbs/2006.04779, 2020.
[52] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforce- ment learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 1094- 1100. PMLR, 2019.
[53] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. CoRR, abs/1 606.60666600000 015.01150,000 2016.
[54] Grady Williams, Paul Drews, Brian Goldfain, James M. Rehg, and Evangelos A. Theodorou. Aggressive driving with model predictive path integral control. In Danica Kragic, Antonio Bicchi, and Alessandro De Luca, editors, 2016 IEEE International Conference Robotics and Automation, ICRA 2016, Stockholm, Sweden, May 16-21, 2016, pages 1433-1440. IEEE, 2016.
[55] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks, 2017.
[56] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning, CoRR, abs/1312.5602, 2013.
13

---

Under review as a conference paper at ICLR 2023
Supplementary Materials for "Model-free Reinforcement Learning that Transfers using Random Features"
A ALGORITHM PSEUDOCODE
Algorithm Model-free Transfer with Randomized Cumulants and Model-predicttvy Control
1: Input: Offline dataset D given by 2.1 distribution p over R number of random features K
2: Offline Training Phase:
3: Randomly sample {0x (QEKK]] with 02 ~ and construct dataset
Do = ((TT
Σ
2h 6(5% ah: [w)) (E(MM)AE]K)
kE(H]
4: Fit random Q-basis functions (b(1,,, vk) over the dataset D
S AH + +R for k [K] by minimizing the loss
[PK]LE[K] E argmin {or becey
Σ
(lsT,aTHiiv
Σ 7 h- 0(5% 60(% Ch 0%))
M
mE(M])AE|K]
kE
5: Online Planning Phase:
6: Fit the testing task's reward function r(.,)) with linear regression on random features:
w" E argmin
1
Σ ME(
(r(8)) ab))
Σ
wrbls% , ax; :w))) + Al|wol12
MH
kE[R]
where rss a%) Reg.aw"
7: Sample 81 pAg
8: for time index / =1 1, do
9: Randomly generate sequences of actions {af, "ar (HenNN
10: Find the best sequence such that
whe Σ
ni E argmax RE[N]
KE[K]
Execute a, from the sequence n,, observe the new state 8111 ~ T(s a;
11: end for
B ADDITIONAL EXPERIMENTS AND SETUP DETAILS
In this section, we provide more details of the experiments, including more detailed setup and sup- plementary results.
B. DESCRIPTION OF ENVIRONMENTS
We describe the details of all used environments such as observation space, action space, reward, offline / online objectives, and dataset collection.
Meta-Worid All our meta-world [52) domains share the standard meta-world observation which includes gripper location, and object locations of all possible objects involved in the Metaworld benchmark. Although the observation space has 39 dimensions, each domain only uses objects so only 7 dimensions are changing any each domain we chose. For pixel observation variants of each domain, concatenate two 84 x 84 RGB images from two views, with resulting
two
14

---

Under review as a conference paper at ICLR 2023
observation dimension of 42336. Each domain has a dimensional action space, corresponding to the delta movement of end-effector in the workspace along with the delta movement of the gripper finger. Metaworld provides a set of 50 goal configurations for each domain. We collect offline dataset following the procedure described Sec. B.2 The online objective is chosen to be a novel configuration that isn't any of the 50 offline goal configurations. To create the privileged dataset, we choose 25 of the goal configurations as offline objectives. These chosen configurations are the furthest 25 goals from the online objective in the Euclidean distance. We evenly annotate the offline dataset with rewards for each of these goals to form a privileged dataset such that the online objective out of the distribution of the offline objectives. For different configurations of the same domain, since object locations are in observation and the goal configuration isn't in it, the dynamics is the same. We now describe the objectives of all used meta-world domains, including those used in the appendix.
1. Reach across Wall The objective is to reach target across wall. The location of the target is randomized for each goal configuration.
2. Open Door The objective is to open the door certain angle by putting the end-effector between the door handle and the door surface. For each configuration, the location of the door is randomized.
3. Turn on Faucet The objective is to turn on a faucet by rotating it along an axis. For each goal configuration, the location of the faucet is randomized.
4. Press Button The objective is to press a button horizontally. The location of the button is randomized for each goal configuration.
Hopper Hopper is an environment with a higher dimensional observation of 11 dimensions and an action space dimension of 3. Hopper locomotion environment that requires long-horizon reasoning since a wrong action will make it fall down and touch the ground only after some steps. The objective of the original Hopper enviroment in OpenAI gym is train to run forward. To analyze the performance of CQL and SF on Hopper, we modify the objective to a goal-conditioned variant such that the agent is trained to follow a certain veloeity specified by its goal. Similar modifications are common in meta reinforcement learning such as in (55]. We sample set of 50 goal velocities between 0 and 1.0 as offline objectives to collect data in the same way as we did in Metaworld environment. For online transfer, we choose four domains with four different online objectives. All the variant domains of Hopper share the same dynamics.
1. Hopper Stand The objective is to stand still and upright. This online objective is the boundary of the offline objectives' range.
2. Hopper Sprint The objective is to run forward distribution of offline objectives ranging from 0 to 1.0. The reward function remains the same as that for offline objectives. Only the goal changes.
speed of 1.5, which is out of the
3. Hopper Backward The objective is to run backward at a target speed of 0.5, which is in the opposite direction of the offline objective. Falling down the ground is penalized.
4. Hopper Jump The objective is to jump to a height of 1.5, without moving forward or backward. Such height typically not achievable when running forward, so this objective is drastically different from offline objectives.
D'Claw D'Claw environment is a dexterous manipulation environment with 24 dimensions of observation space and 9 dimensions of action space. The 9 dimensions correspond to the 9 joints of a robot hand with 3 fingers and joints on each finger. The objective of the environment is to control the hand to rotate rotating tripod to a certain angle. Angular distance to the target angle is used as the offline objective. To increase the degree of freedom and make the environment more challenging, we allow the tripod to not only rotate but also translate freely on the 2d plane. The initial rotation and position of thetripod are randomized upon every episode. We collect the offline dataset in the same way in meta-world training on 50 offline objectives and using e-greedy to collect rollouts. At test time we choose a new offline objective angle and annotate the rewards the privileged dataset in the same way did for goal conditioned Metaworld environments.
15

---

Under review as a conference paper at ICLR 2023
Analytical 2D Point Point is 2D point goal-reaching environment with linear dynamics. The reward is defined as the distance to goal minus an action penalty. The offiine objectives are negative distances to the randomly selected goals on the 2d plane. The online objectives novel goals on the plane. Since we are not evaluating CQL and SF on this environment, we don't generate the privileged dataset for 21) point.
Point Perturbed Point Perturbed shares the same linear dynamics as Point, but features unsafe regions with negative rewards or local maxima with small positive rewards. These perturbations represent out-of-distribution test objectives that cannot be well approximated by low-dimensional feature vector. Note that the added perturbations make point perturbed no longer an instance of analytical 2D point environment with different goal. Instead, it features online objectives that are completely in a different class from 2D point.
B.2 DESCRIPTION OF ALGORITHM TRAINING DETAILS
For each domain, we first train 50 policies with SAC 77. Each policy is trained towards some offline objective of the domain described in Sec. B. for 50000 steps. We then use an c-greedy version of each trained policy to collect 32000 data points for each domain per offline objective. We choose 0.5. Such procedure ensures the dataset has reasonable coverage of the entire state-action space. We note training these policies are fully optional, since RaMP only trajectories without rewards. Datasets collected via intrinsic rewards like curiosity would totally suffice. We choose the random feature dimension to be 2048. Each dimension in the random feature is extracted by feeding state-action tuple to a randomly initialized MIP with 2 hidden layers of size of 32. There are therefore 2048 independent, small random MIPs to extract ø. All state-action tuples are projected reward basis with it.
During offline training phase, we ensemble 8 instances of MLP with 2 hidden layers of size 4096 and train & network following Sec. 3.1 We train e network with learning rate of 3 x 10-4 on the offline dataset for 4 epochs, with a decay of 0.99 and batch size 128. We choose the horizon H be 10 for meta-world and D'Claw environments and 32 for Hopper environments. During online adaptation phase, we first do random exploration for 2457 steps to collect enough data points for linear regression. When doing linear regression, we always concatenate bias dimension to th. For each MPC rollout, we randomly sample 1024 action sequences. We penalize the predicted reward with 0.16 of the variance of predictions from all ensembles. Since online least square makes recomputing regression fast, we perform update of weight vector every single step after initial random exploration finished.
B.3 ADDITIONAL RESULTS ON HOPPER STAND
1e3
Hopper Stand
Due to page limit, we omitted the plot for Hopper Stand in 3 Here we provide additional results of RaMP and base- lines for it in Fig. 4 The result consistent with our anal- ysis in the main paper. RaMP outperforms the baselines just in other Hopper variants. One major difference here is that CQL is performing well for HopperStand. This is likely be- cause the online objective of Hopper Stand is at the boundary of offline objectives described in Sec. B. Given that of- fline objectives are running at target velocities, the CQL likely leams not fall down even if the online objective is out of distribution. By not falling down alone, CQL is capable of maintaining good reward as seen in this case.
Episode Return
2
-1-
-2
0.5
1.0
Environment Steps 1e4
RaMP (Ours)
SF
MBPO
COL
B.4 SCALING
TO
PIXEL OBSERVATION
Figure 4: Results on Hopper Stand
In Sec we evaluate RaMP's 's ability to scale to environ- ments with high dimensional observations. In this section, go step further by significantly increasing the dimension of the observation space to 42336 as described in Sec. Sec. B.I We use a CNN encoder following the architecture in [56] followed by 2 layer MIP as the random feature network.
16

---

Under review as a conference paper at ICLR 2023
1e2
Pixel Reach
Pixel Faucet Open
Pixel Door Open
Egisode Retum
5
2
1-
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Environment Steps
1e4
Environment Steps
Environment Steps
le4
(Ours)
Dreamer
Figure 5: Results on Metaworld with high- -ish-dimensional pixel observation. RaMP achieves compara- ble performance to Dreamer on Pixel Reach and Pixel Faucet Open but struggles to perform well on Pixel Door Open. The performance of RaMP Pixel Faucet Open and Pixel Reach is very similar to its performance in state observation environments.
Table 3: Return as a function of random feature dimension. Low dimensional random features unable to approximate the true reward with linear regression, leading to degraded convergence performance.
Point
Point Perturbed
Metaworld Reach
128
30.46 + 1.62
29.61 + 2.55
765.38 129.98
256
31.53 + 1.45
31.09 + 0.04
806.12 104.77
512
33.02 + 1.10
33.13 + 2.09
824.56 + 121.52
1024
33.36 + 1.10
32.02 2.38
855.14 85.23
2048
34.01
+ 1.04
31.83 + 1.31
830.84 = 142.02
4096
33.31 1.15
32.05 + 1.11
802.45 94.47
Both CNN and MLP layers are randomly initialized. Action projected and concatenated to the first layer of MLP so the random feature would still be conditioned on both action and observation. We compare our method against the Dreamer [19, the state of art model-based learning method optimized for pixel observations. Similar to MBPO, we pre-train dreamer's s dynamics branch with offline data before the online phase. As shown in Fig. 5 our method able to achieve a similar level of performance as Dreamer in two meta-world environments, Pixel Reach and Pixel Faucet Open. RaMP does not see a sigmificant return drop from the variant the environment with state observation. Given the efficacy of Dreamer, the result still shows RaMP'S performance can scale up to pixel observation. However, Dreamer is able to outperform RaMP significantly in an environment like Pixel Door Open. This likely because random features capture the change in input space rather than reward space. Pixel observations can give random features a lot of noise while important semantic features may correspond random convolution layers, we can use rre-trained encoders to achieve better semantic feature ex- traction and significantly improve the quality of random features. This is beyond the scope of our work and we leave this for future works.
big change in pixel space. We note that instead of using
B.5 ADDITIONAL ABLATIONS
Our method builds on the assumption that a linear combination of high dimensional random features approximate arbitrary reward functions. In the following ablation experiments, we validate this assumption both quantitatively and qualitatively.
Effect of random feature dimension We evaluate method on Point, Point Perturbed, and Meta- world Reach using {128, 256, 512, 1024, 2048, 4096] random features. Results are summarized in Table We find that performance degrades with smaller random feature dimensions because the features are unable to linearly approximate the true reward. On the other hand, Fig. 5 shows that high dimensional random features experience slower adaptation.
17

---

Under review as a conference paper at ICLR 2023
1e1
Point
Point Perturbed
1e3
Metaworid Reach
1.0
0.8
2
0.4
0.2
-2
0.0
0.5
1.0
0.0
0.5
1.0
0.5
1.0
Environment Steps
Environment Steps
Environment Steps
1e4
Figure 6: Learning curves for different random feature dimensions. Low-dimensional random fea- tures suffer from poor convergence performance, whereas high-dimensional random features expe- rience slow adaptation.
1e2
Button Press
1e3
Hopper
1e2
D'Claw
Episode Return
1.5
2
1.0
0
-2
0.5
0.0
0.5
1.0
0.5
1.0
0.0
0.5
1.0
Environment Steps
1e4
Environment Steps
1e4
Environment Steps
1e4
RaMP (Ours)
MBPO
SF
SF In Dist
COL
COL In Dist
Figure 7: In-distribution results for Successor Features and CQL results. Note that RaMP and MBPO unaffected since they do not depend the distribution of the oflline objectives.
Scaling with state dimension In Table we evaluate method on point reaching environments with 2. 3, and 4 state dimensions. All three environments feature distance-to- go-goal minus action penalty as the reward. We compute the return eтror stemming from linear regression as well as the Q error stemming from both linear regression and function approximation. We see an increase in both return error and Q error with higher state dimensions, but the overall approximation errors remain reasonably low.
Nonlinear approximation capability In Fig. we visualize the truncated Q value obtained from lin- ear combination the random cumulants and com- pare it to the ground truth Q value approximated by Monte-Carlo sampling. We perform the comparison with Point and Point Perturbed environments. Our method provides an accurate estimate of the Q value even in the face of out-of-distribution and highly nonlinear rewards.
True Q Values
Predicted Q Values
50
Point
Finally, compare the performance of RaMP baselines on in-distribution online objectives. Our method is designed make no assumptions about test objectives during policy transfer. As shown in Fig. 3 RaMP outperforms CQL and SF when the online objectives are out of distribution. A natu- ral question to ask is how things will change when the setting satisfies the assumptions of offline-online objective correlation. For example, a 2D reach- ing enviromment, the training dataset may be anno- tated with either rewards corresponding to only goals on the right half goals covering the entire plane.
Point Perturbel
-50
Figure Visualization of true value and approximated Q value. Our method is able to approximate the Q value in the face of out- of-distribution and highly nonlinear rewards.
18

---

Under review as a conference paper at ICLR 2023
Table Approximation error with different state dimensions. As state dimension increases, approx- imation error increases but remains in reasonable range.
Point 2D
Point 3D
Point 4D
Return Error
0.42 + 0.20
1.20 + 0.61
1.32 + 0.70
Q Error
0.32 0.05
0.51 0.10
0.61 + 0.12
1c2
Reach
le3
Reach Wall
1e2
D'Claw
0.8
0.6
0.2
0.00
0.25
0.50
0.75
1.00
1.25
0.00
0.25
0.50
0.75
1.00
1.25
0.00
0.25
0.50
0.75
1.00
Environment Steps
lef
Environment Stops
1e4
Environment Steps
1e4
Random Shooting
MPPI
Figure 9: MPPI results on World and D'Claw MPPI improves the performance of our method across all four tasks, showing that our method can benefit from powerful planners.
When the online objective is to reach goal on the left half of the plane, it will be of distribution for the first case while being in distribution for the second. When we curate the labeling process of the privileged dataset to satisfy the in-distribution assumption, CQL and SF receive a significant per- formance boost. As shown in Fig. the performance of our method and MBPO are unaffected as neither algorithm depends on offline objectives. CQL, on the other hand, matches the performance of our method under this new setting. This serves foil to the generalization of our method to -istr--disttiiution online objectives.
B.6 MPPI EXPERIMENTS
We provide additional results on model-predictivy control via model- preedc-predictvve path integral (MPPI) [54] in Fig. 9 MPPI is sampling-based trajectory optimization method which maintains distri- bution of action sequence initialized as isotropic standard Gaussian. In each iteration, MPPI draws action sequences from the current distribution and computes their values, which we do using the learned Q-basis networks and online regression weights. MPPI then updates the distribution using the weighted mean and standard deviation of the sampled trajectories, where the weights are com- puted as the softmax of the values multiplied by temperature parameter ?. As shown in Fig. 9 MPPI improves the performance of our method across Metaworld environments and the D' C'Claw environment, thus indicating that our method can benefit from powerful planning algorithms. In these experiments, we perform optimization steps and sample n 1000 trajectories each step. We use 10 for Metaworld and 50 for D' D'claw.
B.7 FINETUNING EXPERIMENTS
While we freeze the Q-basis networks at test time in our main experiments to demonstrate transfer behavior, we can fact finetune the Q-basis networks to continuously improve our estimate of the Q-value. After performing reward regression for number of steps, can finetune the Q-basis networks on online trajectories by fitting the predicted Q Q-values to the Monte-Carle Q values and allowing the gradients flow through the regression weights. We conduct finetuning experiments on two Metaworld enviromments and the D' Claw enironments. As shown in Fig. T0 our method sees noticeable performance increase with finetuning starting 6400 steps.
19

---

Under review as a conference paper at ICLR 2023
Reach
Reach Wall
D'Claw
1.0
Episede Return
0.8
0.8
2-
0.4
0.2
0.2
-2
0.00
0.25
0.50
0.75
1.00
1.25
0.00
0.25
0.50
0.75
1.00
1.25 le4
0.00
0.25
0.50
0.75
1.00
Environment
led
Environment Steps
Environment Steps
le4
Finetuning
Finetuning
Figure 10: Finetuning results MetaWorld and D' D'Claw. Our method is able to continuously improve by finetuning the Q-hasis networks during online training.
C DETAILED THEORETICAL RESULTS
In this section, we provide the formal statement of the theoretical insights given in $3.3 and corre- sponding proofs.
C.I FORMAL STATEMENT
Theorem C.1. Suppose the offline data in D are generated from some distribution p E A(S x A),
Le., (s%, ax) +) and 5/41 T for all (m,h) E [M] [H], and inf ((s, 2 0. (s,a)ESxA
Suppose O2 p(-) for all k E [K], and
sup Ax.
(08,a;0) 0) S K. for some > 0 and
:))) is continuous. For some large enough MH, letting À n-1 we have that if K ((Vn (og(mz/(///))) then with probability at least 6, for any given reward function R
(1²) - Q2lloo S
1
inf JEH
Σ
/ (r (r- (r..)))) +o log (og(1/8) vn
1-
p
(s.0)ESXA
E(
for any policy т,, where H. = {f = ((((=))u()))p(0)) lw(())ppp(0) < oo}, w" is the solution to (3.2 and
Qn(s,aw"))
widlsh: ani an;0k) = kE(K]
(C.1)
The proof of Theorem dom features, the Q- Q-function of any reward function R, under any policy π,, can be approximated accurately up some inherent error related the richness of the function class that the features can represent. Note that we here only state the results under some mild and basic assumptions from the random features literature, and are by no means tight. They can improved in various ways, for example, if the sampling distribution of 0, p, can be data-dependent, and some stronger assumptions on the data and kernel function classes 47 50].
be found in
shows that with large enough amount of ran-
Corollary C.2. Suppose the assumptions in Theorem C.I hold, and additionally the kernel induced
function space H. rich enough such that inf E(F 0. Then, with horizon length H
O log( (Raux/ and M = Ô (((-))=))) episodes in dataset D, and with K Ōi(1 - - -2)
1-7
random features, we have !! (w²) 11 < O(e) for any where for each (s, a). Q (s, w²)
a H -hrrionn truncation (C.T. which can be estimated from the offline dataset D.
The proof of Corollary C.2 can be found in C22 which specifies the number of random features, the horizon length per episode, and the number of episodes, in order to approximate Q= accurately
20

---

Under review as a conference paper at ICLR 2023
by using the data in D. Note that the number of random features is not excessive and is polynomial in problem parameters. Combining Theorem C.I and Corollary C.2 leads to the informal statement in Theorem 3.1
Next, we justify the use of open-loop Q-functions in planning in a deterministic transition environ- ment, which contains all the environments empirical results have evaluated. Recall that for any given reward R, let 2₂ denote the Q-function under policy я. Note that with a slight abuse of notation, Qn can also be the multi-step Q-function (see definition in $2), and the meaning should be clear from the input, i.e., whether it is Q~ (8, a) or Q= "1.............................. ).
Theorem C.3. Let II be some subclass of Markov stationary policies, i.e., for any 11, S + A (AA). Suppose the transition dynamics T is deterministic. For any given reward R. denote the step policy obtained from H H-step open-loop policy improvement over II as n'H S - A" defined as
TH(s) E
argmaax max Qu a1, ,aH),
(01"".00)) ( TEll
for all S. Finally, define the value-function under T as V (s) == Quy (8, x (s)). where Qry (s, (1:H) is the fixed-point of the Bellman operator Th, defined (C.8). Then, we have that for all S
Vaja (8) 2 max max Q=(s,11,". .an) 2 max max Qx(88 a). nell
#1:8 nell
C.2 DEFERRED PROOFS
C.2.1 PROOF OF THEOREM
The proof relies on the result of generalization guarantees of learning from random features, with squared loss. Note that one cannot use the results in (46), which dealt with Lipschitz loss function of the form c(y', y) = c(y'y) y). This does not include the squared loss we used in our experiments. Instead, we resort to the results in (47). which also yield better statistical rate. For the sake of completeness, we re-state the abridged version of one key result therein, Theorem in [7], as follows.
Lemma C.4. Suppose that K is a kernel with an integral representation K x') Sa (ax, w)b(a' w)dp(w), where (2, probability space and vb X x Ω - IR, where X separable space. Suppose vp is continuous and [ı(2,, w)l s with +oo) almost surely, and |y| S almost surely. Define the expected risk:
E(f) = / (f(x) y) 1)/dp(x,1)) y),
where p is the distribution where the data samples v:) Define the solution kernel ridge regression with M random features as
(x) = ths (x) x with Wx.M = SM + -
(C.2)
where der (x) = (v(z, w1) 1b(,, w2). (ax, WMs))/ /MM w; are drawn i.i.d. from p(-), 77 (Ur.". ... 3.n) /22 Ste == (D ... (xr(xn) 1/2 Then, suppose n 2 no. X = 1/n and the number of features M 2 soyilog(10Sk 2 vn/5), we have that with probability at least 1 6,
E(..M) - min E()) < Ci log (18/6)
JER
Vn
where no, cu. Cj are absolute constants, H. is the reproducing kernel Hilbert space corresponding to the kernel K.
We then apply Lemma C.4 with (a, y) in the lemma being replaced by ((s, a). r(s, a)).
p(., y), P. x. w. M, in the lemma being replaced by p(s,a) R P, (s, a), O. K, X in our case.
21

---

Under review as a conference paper at ICLR 2023
Note that Lemma C.4 requires the space X to be separable, and our finite space S A satisfies; it re- quires | bounded, and our reward absolutely bounded by Rmax, and thus also satisfies. We hence obtain that with probability at least - 8, if the number of random features K' 2 22 lo vn)), with n = HM, then
inf E(f) + O log(1/0) FEH
E (r
r-
Σ KE(K]
wi
Vn
(C.3)
where we note that w (wi,," 1 , is the solution to Theorem C1
3.2) and the E()) here is defined in
For any policy a for the MDP, let Q+ denote the Q-function under policy and the actual reward function distribution R, and (="")) denote the Q-function under the estimated reward using ran- dom features:
(,(s,a;w))== = E Σ (shh on; w²) 81 8,
where F(s,a,w))
ak ak))
(C.4)
kE[K]
By Bellman equation, we have that for each ()
-u rdd
8'
=(),a;"")) - "0 (T('
8'.a'
- (ss,,;;;")) +. $ $1ʃ f (
+y. 10. Qn(wm)
Taking sup over a and organizing the terms, we have
Qx - (u)")
|| $ 1-7
supppp sup / rddRaa()) (Fl)a((())F(s,a;)")))) s;a
(r) F(s, - ((s,;;w0)) 2
Σ
(C.5)
1-7
8,a
2 (r) F(8, a; wx) - p(s,a)
(C.6)
1 1
V
8,0
(1 12 V E(s.a)olss) (
(C.7)
where C.5) uses that 1 - |l - ll2 for mnite-dimensionll vectors, (C.6) uses the definition of p- Further, by Jensen's inequality, for each (s,a)
( (JrdR..o (r) F(s,, (ls.aa"")) (EnwR O Or F(s, F(s,a s BI F(s, a; w²
which, combined with C.7 and C.3) gives that
Q @x(10))
< (1 1) V ((aa)))tts))) Rmal) (ʃ (rdRsalr)) - F(s, wr)
E() + 0 108(1/0) SEN
< "1- which completes the proof.
V
inf
Vn
22

---

Under review as a conference paper at ICLR 2023
C.2.2 PROOF OF COROLLARY
First, note that with II O log Rmmxx (c(2->)))) >))) ensures that 11 (w+) (1l)-Qree (ll I O(e),
1-7 which can be obtained by the boundedness of r(s, a) by Rmax. and the fact that
lo Riaxe [C(( (c(1-))))
Rmax (1- (1-))- 7)) 111
Rmax
Remax
1-7
Furthermore, since Theorem C.1 requires HM = 0 (1-3)<0))) to make sure 11Q+(w")
Qx S Combining these lacts yields the desired result.
C.2.3 PROOF OF THEOREM 3.2
Define
(8,01, "1, ,aH) := max (s,00,, (I,,"",,,,)))) and (8, == max Qx(s,a). se1l
#ell
We also define the Bellman operator under the open-loop policy ThH follows: for any Q ||S|x|441 TH (Q)(s,01, ,a)) = Σ hE[H]
7th r(sh, an) + "(s5441 81 s, 01:H
(C.8)
Note that TH,, is contracting operator, and we denote the fixed point of the operator as QII.TH ||S|x|4*1 which is the Q-value function under open-loop policy Ty.. By definition, we also know that the state-value function under THH Vit,Hh = =m (s, ( (s)), i.e., by applying the open-loop policy ThH the actions 01:H Q H,rk, (8, a1:H). Note that
T (Q ( (8, (8,,11,"" .a.)) E Σ
nh r(sn, an) " (3
01:1
KE(H
=
E
rr an) , max (SHH1,
s, 3,61
(C.9)
684128
he(H]]
2 E
7 h- h- +
max 2:(s411.041:22
81 = 8, 01.H
(C.10)
auti
he(m)
2 E
Σ
hー h-1 ((s, an) + 2Q, ((HH) ((8==)...,
81 = 8, G1.H
(C.11)
=Q (1, aH),
(C.12)
for any E II, where C..9 uses the definition of Ttz- (0.10) is due to the definition of Qmax, and
(C.IT) is by the max 11.8 and (C.12) is by definition. Since (C.12) holds for any क IT, by the monotonicity of TH we have
Qu (8, (1,)
lim (TH,,,,) ((Qaa))))))))))))))))))))))))))) (s, (s,,)..)) 2 Q 1-000
(8, (8, an.)) 2 max Q=(s,an.1)) a1:H).
sell
(C.13)
Notice that for all s, by applying (s) on both sides (C.I3).
VH (s) QH (s, T'H (s) 2 max Qx (s, THH max max Qa (s, (1:H). d well
(C.14)
#ell
Further, due to the multi-step maximization, we have
max max Q= ".,... ,nn) max max (x a),
#1:8 well
which, combined with (C.14 completes the proof.
23

---

Under review as a conference paper at ICLR 2023
D RELATIONSHIP TO EXISTING WORK
We briefly connect our proposed algorithm to prior work.
Successor features for transfer in RL: While successor features 40
44 have shown the ability to transfer across problems in RL, the two key differences our framework are (1) using random features rather than requiring learned pre-provided features and (2) training open-loop Q functions rather than typical Qx- These two changes allow transfer to happen across a broader class of reward functions and not simply be restricted to the policy cover experienced at training time.
Our work is connected to model based RL in that it disentangles dynamics and rewards, but is crucially different in that it doesn't model one-step evolution of state but rather long term accumulation of random features. This trades off compounding error for generalization error.
Model-based RL:
Our work connected to methods for model-free RL in that also models a set of Q f-functions, but importantly this doesn't 't correspond to particular reward, but rather random features of state. By doing so, we are able to adapt to arbitrary rewards at test eest-time, rather than being tied to particular reward function.
Model-free RL:
E INFINITE-HORIZON Q-FUNCTION VARIANT
While our setup in Section 2 uses finite-horizon Q to approximate Q=, our method can also plan with an adfnite-horizon Q function during the online phase. In this section, we describe one compatible way to learn an infinite-horizon Q-function while still enjoying the benefits of RaMP in the case with deterministic transition dynamics. We also present empirical results and analysis of this variant.
E. METHOD
We first notice that an infinite-horizon Q-function Qn can be decomposed into the discounted sum of H-step reward and discounted value function under policy क evaluated at st+
Qn =E
7 Vv (SH++))
1- r(s,, az) s', a'
arrorf- (82) S+11T(( s()a,)
1=1
where Va(s') = Earae (x))
Σ x (r1, =
Given a policy æ, value function V parameterized by 0 can be learned via gradient descent and Monte-Carlo method:
0+0 - V [ - (rr + VV 11 for sampled (Sur4 dey re) ~ Tn
where Tx is trajectory rollouts collected with current policy and 0' is target network that gets updated by 0 with momentum.
Now consider our multi-step setup. Our multi-step Q-function can also be written as the sum of our H step approximation and discounted value function 8 8H+1
T1.H) = Ban-roT(- (8++)) 111 (158))))
re 91 =, a..... .a = ēH
Q (s,ā1:H) + V1 (sH+1),
where note that in the last line, there is expectation over 8141 since the transition dynamics deterministic, and 8H+1 is deterministically decided by (s1, 41:H).
Vanilla RaMP enables efficient estimation of Qx with novel reward function the cost of truncating the second term above with Q= se Q As we have shown Section 4 planning with this finite- horizon Q-approximation would already lead to reasonable planning in most of the experiments.
24

---

Under review as a conference paper at ICLR 2023
We can go step further and also estimate the second term can plan in setting. The main challenge is getting Va ($ in our multi-step setup. as we don't explicitly predict sH++1. This, however, can be addressed easily by reparameterizing (=(===1) on an action sequence that leads to 81++ just like what we did for Q. We thus define multi-step value function
(=,8.11.1 Eag (1
5,41 ū1 an u
Then Q (s, (1.11) = Q= (1s.)..)++++ +?" F (s, d1:H). Under our deterministic transition dynamics, 81++ 1 fully determined by (81, (1:11) ) so we can remove the expectation in the equation. We then rewrite the training objective V= in terms of F to learn Fx:
- - V (*+4) - (r44 + VV (844++1)) for sampled (s=+Hic+H+1:0 Hc F(tH)
becomes
0- Vo () (14+H-1)
(r++# FF (st+1, (411+4))]
for sampled (5:t+1100tt+ Tt+ Tm
where parameterize multi-step value function Fx by 1t So we can learn Fr with Monte- Carlo sampling and gradient descent just like what we do to learn V- in the single-step case above. Combined with ! we now have an estimation for the fffnite-horizon Q-function Q in a multi- step manner.
For planning. we do on policy learning by alternating between policy rollout and Q learning. policy, MPC planner first uses the infinite-horizon (r 01:H (I) = Q# (s, 01.H1 + - F to plan and collect rollouts. Then Fs is trained with these on policy rollouts while Q is also learned like vanilla RaMP via online least square. By incorporating this aifiite-horizon variant, MPC planner can now plan with an infinite horizon during the online phase.
(1:H1)
E.2 EXPERIMENT AND ANALYSIS
We implemented the infinite-horizon variant described above and carried out experiments to quan- titatively evaluate its effect on performance four environments with varying types tasks. As shown in Figure II we found that the infinite-horizon variant out-performs vanilla RaMP with finite horizon Q-function in Hopper Jump and achieves comparable performance in D'Claw, both being the longer horizon environments among the four. However, the aufiitte-hoiionn variant actually performs worse in two Metaworld manipulation environments, ReachWall and FaucetOpen, likely because these tasks do not require function V has to be learned from scratch from samples, which may even hurt the performance at the beginning of the online phase.
reasoning. In particular, the multi-step value
Reach Wall
Faucet Open
Hopper Jump
D'Claw
Epicade Retur
2
-2
1
Environment Steps 184
Environment Steps 1o4
Environment Steps 104
Environment Steps 1e4
Finite Horizon
Infinite Horizon
Figure 11: Results of RaMP variant with infinite horizon Q-function on four different environments. We notice this variant leads to better performance on longer-horizon horizon tasks but doesn't increase the performance of metaworld tasks, likely because the benchmarked tasks don' 't require infinite-horizon reasoning. Since the V-function has to learned from scratch, it could also hurt the performance at the beginning in low data setting.
We also present results this variant with anfiite-hhorizon Q- Q-function on all 8 environments com- pared with all baselines. We run all the environments for 25600 steps, at twice the amount of steps
25

---

Under review as a conference paper at ICLR 2023
did in Section 4 We also changed the discount factor 7 to 0.9 to emphasize the effectiveness of RaMP's quick adaptation to new reward using Q" As shown Figure 12 RaMP has the ability to quickly adapt to sees more data, the advantage of RaMP is not salient as our infinite-horizon value function learning using vanilla bootstrapping. However, since the infinite horizon-variant uses bootstrapping, the method can continuously improve like all RL methods as samples grow. This will be useful for longer horizon and harder tasks.
reward function, when the amount of data is extremely low. As the algorithm
We note that although we used the simplest bootstrapping approach to learn multi-step V, we can use any other value estimation method make it more efficient. The core of this variant to parameterize the value function at the H + 1-step with the initial state and sequence of actions.
1e2
Reach Wall
1e2
Door Open
1e2 Faucet Open
1e2
Button Press
2.0
Retur
15-
3-
Hopper Sprint
1e4
Hopper Jump
1e4
le3Hopper Backward
led
D'Claw
1e4
Episede Return
1,00
-2
0.75
-4-
0.25
-6-
-2
0.00
Environment Steps le4
Environment Steps 1e4
Ervi dewironment Steps le4
Environment Steps le4
We benchmark RaMP with infinite-horizon Q-function all environments for more steps. RaMP can quickly adapt to new test rewards in low-data regime and can continuously improve with the addition of bootstrapping. However, it will need to rely more-sample efficient value function learning methods to be more sample efficient in the infinite-horizon setting. This shows RaMP's benefits largely lie in the low-data regime
Figure 12:
F COMPLEXITY ANALYSIS
The space complexity of RaMP primarily determined by the number of random features that are needed. As we describe in Corollary C.2 we require K = π²(( - - random features to achieve e-order
between the estimated and Q-function. As the required approximation emror decreases, the space needed to store all the features grows sublinearly. Note that this result is for any given reward function (including the target one) under any policy æ,, but not tied a specific one (under the realizability assumption of the reward function).
On the other hand, the space complexity of the classical successor feature method relatively fixed in the above sense: the dimension of the feature is fixed and does not change with the accuracy of approximating the optimal Q-function for the target task. However, the resulting guarantee is also more restricted: it is only for the optimal Q-function of the specific target reward, and also depends on the distance between the previously seen and the target reward functions. Hence, the two space complexities are not necessarily comparable.
26


Reviews:
[{"review_id": "wye1VvO8ow", "reviewer": "Reviewer_5Avm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to model long-term behaviors for random cumulants, or Q-basis functions, in an open-loop manner. At test time, they can solve a ridge regression problem to fit the coefficient vector for a new task given some environment samples. They present the MPC-based planning, which can be used to gather samples. The authors also suggest the theoretical background on the relationship between the approximation of Q-functions and the used number of features and the horizon and the number of episodes. They show the experimental results in multiple manipulation and locomotion environments.", "summary_of_the_review": "I find the proposed work reasonably novel, and the presented theoretical results are helpful for the community. They provide meaningful empirical results to show the effectiveness of the proposed approach. The writing and presentation is good.", "strength_and_weaknesses": "Strengths\n\n- Despite its similarity to successor features, the authors successfully present how the proposed method is different and what its advantages are compared to successor features.\n- Thm.3.1, wihch incorporates multiple hyperparameters, provides the theoretical background and justification to the proposed method.\n- The overall presentation (especially Fig.1) and writing is good and easy to follow.\n\nWeaknesses\n\n- It would be informative to see how each method that appears in the comparison does well for different areas of goals, because the manifold of the used reward functions could affect their performance.\n- Analyzing the space complexity of the method, especially in comparison with prior methods (mainly successor features) would be useful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ryNoGq7w8_a", "reviewer": "Reviewer_xLea", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach to model-free transfer learning in RL called Random Features for Model-Free Planning (RaMP). RaMP works by learning fixed horizon open-loop value functions (predicting the total reward received if H specific actions are selected in sequence starting from a state) for each of k randomly generated reward functions. When a new problem is encountered the algorithm learns to predict the rewards as a linear combination of the k random rewards via least squares. The fixed horizon open-loop action values for the new problem are likewise estimated as a waited combination of the k-learned value functions using these least squares coefficients, leveraging the linearity of fixed horizon open-loop value functions. These value function estimates are then used to select an action in the new environment by random shooting, that is randomly generating N action sequences and selecting the best according to the value function approximation. RaMP is experimentally compared in simulated robotics domains to a number of other transfer learning approaches, including model-based MBPO.", "summary_of_the_review": "The method presented in this paper is potentially interesting as a reasonable baseline for transfer learning in certain problems. Further work in this direction has the potential to produce something even more interesting and broadly applicable. However, I believe that in the current version of the paper there are serious issues with the presented theory. In addition, I feel the generality of the method is overstated as the potentially detrimental effects of stochasticity are not discussed. It's not clear that the empirical conclusions are very robust as nothing is said about how hyperparameters are chosen for different algorithms and certain experimental details are unclear.", "strength_and_weaknesses": "Strengths\n=========\nThe method presented in this paper is quite simple and interesting as a reasonable baseline for the transfer RL setting. The idea of learning action values for a large number of random reward functions as a tractable route towards transfer to new environments is intriguing.\n\nFor the most part, the proposed method is coherent, reasonable and well-described.\n\nThe writing of the paper is generally good.\n\n\nWeaknesses and Uncertainties\n============================\nI am not sure Theorem 3.1 is correct. In particular, there is no stated assumption on the distribution of the random features either in the main body or in the appendix. In the trivial case where all the random features are the same, clearly, the result would not hold as drawing multiple copies of the same feature would not help to represent the function any better. The proof in the appendix apparently uses Theorem 1 of Rudi and Rosaco (2017), however, that result is specifically for random features which approximate some Kernel, which I can't see is the case here, so it's not at all clear to me whether this result is relevant or applicable. Perhaps the authors or one of the other reviewers can clarify this as I am not personally very familiar with all the relevant work.\n\nI also have doubts about Theorem 3.2, especially the part that states that the value of following an open-loop policy is greater than or equal to that of a one-step policy. In the case where $\\Pi$ is the set of all possible policies, it contains the optimal policy, which cannot be improved upon. On the other hand, it is clear that with stochastic transitions there are cases where committing to an open-loop policy for more than one step is detrimental as it deprives you of the ability to choose actions in response to future random transitions for some horizon. The first inequality of the theorem appears to follow immediately from the definition of $V_{\\pi'_H}$ and can be strengthened to equality unless I've misread something, so I also don't see how this part serves to justify the use of open-loop Q-functions in planning. Perhaps the result is meant to apply to deterministic transition dynamics only, but this is not stated anywhere and the background section makes it clear that stochastic transitions are considered.\n\nIt seems like all the displayed results are for a fairly low number of environment steps (around 10,000, see Figure 3), and it appears that in a few cases MBPO may surpass the performance of the proposed method given a little more time. While there is no inherent problem with arguing that the proposed method is useful in the low data regime, it would nonetheless be good to show how the different approaches compare given more updates. Furthermore, no details are given about how the hyperparameters of the approaches are selected which is crucially important in understanding how these results should be interpreted. Especially in the low data regime, things like the learning rate will have a large effect on the performance of different approaches after a fixed number of steps. In the same vein, I wonder if the performance of MBPO could be improved simply by adding more model rollouts.\n\nIf my understanding is correct, this approach selects action sequences which maximize reward only for the duration of the action sequence itself (fixed to 10 for metaworld and 32 for hopper). This is limiting, as one would have to explicitly predict the sum of reward conditioned on any possible action sequence for the entire horizon. Moreover, there seems to be no easy way to generalize the method beyond these circumstances, since any maximization in the random Q-functions would break the linearity which allows one linearly map rewards to associated action-sequence values, however, the paper does explain this point fairly clearly. For the same reason, any significant amount of stochasticity has the potential to break the method, which I didn't find was clearly explained in the paper, and in fact, the theory discussed above seems to incorrectly imply the opposite.\n\nI'm a little confused about exactly how RaMP learns the reward function given a new environment. In appendix A it seems to be suggested that it queries the new reward function on the existing transition data from the offline dataset in order to gather data to train its reward prediction. If so, this is not really clear and would seem to provide RaMP with a potentially unfair advantage if the baselines are not allowed to do the same. This needs to be better explained in any case. Furthermore, I'm not entirely sure why RaMP's performance even changes over time in the online planning phase since according to the pseudocode in Appendix A, all the learning happens before this. This too should be clarified.\n\nMinor Comments, Questions and Suggestions\n=========================================\n* Section 2.1: \"we can enable very quick (even zero-shot) adaptation to the new reward functions encountered at test time\", what does \"zero-shot\" mean in this context?\n* Section 4.3: \"jumping and running backward have drastically different objectives that are difficult to express as parametric 'goals'\", could you elaborate on this? I don't understand how it is the case, especially how running backwards is any harder to parameterize this way than running forward.\n* Section 4.3: \"MBPO fails to match the performance of RaMP since higher dimensional observation and longer horizon increase the compounding error of model-based methods\", this is unclear to me and seems like an unproven claim. As mentioned above how do we know that MBPO couldn't be improved by for example adding more model rollouts.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TvJyxsICNSk", "reviewer": "Reviewer_puvd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to learn the discounted sum of random features over finite horizons using offline trajectories. By using finite-horizons, the authors break the dependence of the Q function on $\\pi$. Once this pre-training phase is done, the agent must learn to linearly combine these discounted sums to learn the new reward function. With this approach, the agent can do model-predictive control to act on the environment.\n", "summary_of_the_review": "The authors present an interesting idea. However, the paper has major gaps and I can't recommend acceptance.", "strength_and_weaknesses": "Strengths\n- Authors propose a new way to exploit off-line trajectories without the need of reward labels.\n\nWeaknesses\n- The finite horizon fixes the dependency of the Q function on the policy. However, it makes longer time horizon tasks intractable. This is a major weakness of the proposed approach.\n- Trajectories are generated with random shooting. It's unclear how this would scale to continuous action spaces or longer horizon tasks.\n- The main results (Figure 3) do not show a clear picture of the results.\n  - How good are the reported results? That is, what would the score of a standard algorithm like PPO be in these cases (given enough training data). Obviously, this is not a 1 to 1 comparison, but it is important to know if the reported scores correspond to almost optimal policies or if the agents just very quickly learn a simple policy that is far from good.\n  - What happens if training goes on for longer? MBPO is quickly improving in 5 of the 8 plots and it seems that if the plots were just a bit longer, the results would be much more positive for MBPO. \n  - RaMP seems to converge quickly and not improve anymore. This seems reasonable given that only the linear combination of Q-basis is learned. Can the learnt policy continue improving somehow? If the proposed approach quickly learns a degenerate policy but it can't improve afterwards, it's unclear the proposed approach would have any applications (unless the point RaMP converges to is already very good).\n- It's unclear that any reasonable number of random features will make the approach work in high dimensional spaces. The only experiments in this setup are in the Appendix and are not very convincing. Why do you not make a comparison to MBPO, SF or other offline RL approaches in this setup?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Rov4ZTemlPK", "reviewer": "Reviewer_R3oi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method of transfer based on the assumption of common dynamics and variable reward signals. The method leverages artificial pseudo-rewards to span a space of probably $Q$-functions related to the down-stream tasks.\n", "summary_of_the_review": "The problem studied in the paper is well-motivated and important. The authors develop a method, which takes advantage of the factorization of the dynamics and rewards. The premise is that the dynamics can be transferred and adaptation to a particular reward structure can be achieved relatively fast. \n\nThe authors propose an implicit method of encoding the model, via random cumulants in order to avoid the usual nuisance of predictive model-based approaches, like compounding errors. This, however, leads to the question, which is my main concern. The approach assumes training a multistep $Q$-function, $Q(a_1, \\ldots, a_H)$ -- n the paper, $H\\in \\{10, 32\\}$. I have doubts if this can be done efficiently for larger $H$, namely, when the long-term credit assignment is  needed. I suspect that either the method requires rather dense and informative rewards.\n\nQuestions:\n - Could the authors comment on implicit assumptions for the method to work (e.g. if I am right with the above concerns)? What kind of problems do have such properties?\n - Have the authors tries other optimization methods, e.g. cross-entropy method.", "strength_and_weaknesses": "Strengths:\n- well-motivated and important problem\n- sound method \n\nWeaknesses:\n- scalability/applicability\n- empirical part ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject