SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where we note that in the last line, there is no expectation over  $s_{H + 1}$  since the transition dynamics is deterministic, and  $s_{H + 1}$  is deterministically decided by  $(s_1,a_{1:H})$ .
Vanilla RaMP enables efficient estimation of  $Q_{\pi}$  with novel reward function at the cost of truncating the second term above with  $Q_{\pi} \approx Q_{\pi}^{H}$ . As we have shown in Section 4, planning with this finite-horizon  $Q$ -approximation would already lead to reasonable planning in most of the experiments.
We can go a step further and also estimate the second term so we can plan in an infinite-horizon setting. The main challenge is getting  $V_{\pi}(s_{H + 1})$  in our multi-step setup, as we don't explicitly predict  $s_{H + 1}$ . This, however, can be addressed easily by reparameterizing  $V_{\pi}(s_{H + 1})$  on an action sequence that leads to  $s_{H + 1}$  just like what we did for  $Q$ . We thus define a multi-step value function
$$
F _ {\pi} (s, \widetilde {a} _ {1: H}) = \mathbb {E} _ {s _ {H + 1}} \left[ V _ {\pi} \left(s _ {H + 1}\right) \mid s _ {1} = s, a _ {1} = \widetilde {a} _ {1}, \dots , a _ {H} = \widetilde {a} _ {H} \right].
$$
Then  $Q_{\pi}(s,a_{1:H}) = Q_{\pi}^{H}(s,a_{1:H}) + \gamma^{H}\cdot F_{\pi}(s,a_{1:H})$ . Under our deterministic transition dynamics,  $s_{H + 1}$  is fully determined by  $(s_1,\widetilde{a}_{1:H})$ , so we can remove the expectation in the equation. We then rewrite the training objective  $V_{\pi}$  in terms of  $F_{\pi}$  to learn  $F_{\pi}$ :
$$
\theta \leftarrow \theta - \alpha \nabla_ {\theta} \left\| V _ {\pi} ^ {\theta} \left(s _ {t + H}\right) - \left(r _ {t + H} + \gamma V _ {\pi} ^ {\theta^ {\prime}} \left(s _ {t + H + 1}\right)\right) \right\| _ {2} ^ {2}
$$
for sampled  $(s_{t + H:t + H + 1},a_{t + H},r_{t + H})\sim \tau_{\pi}$
becomes
$$
\theta \leftarrow \theta - \alpha \nabla_ {\theta} \left| \left| F _ {\pi} ^ {\theta} \left(s _ {t}, a _ {t: t + H - 1}\right) - \left(r _ {t + H} + \gamma F _ {\pi} ^ {\theta^ {\prime}} \left(s _ {t + 1}, a _ {t + 1: t + H}\right)\right) \right| \right| _ {2} ^ {2}
$$
for sampled  $(s_{t:t + 1},a_{t:t + H},r_{t + H})\sim \tau_{\pi}$
where we parameterize multi-step value function  $F_{\pi}$  by  $F_{\pi}^{\theta}$ . So we can learn  $F_{\pi}$  with Monte-Carlo sampling and gradient descent just like what we do to learn  $V_{\pi}$  in the single-step case above. Combined with  $Q_{\pi}^{H}$ , we now have an estimation for the infinite-horizon  $Q$ -function  $Q_{\pi}$  in a multi-step manner.
For planning, we do on policy learning by alternating between policy rollout and  $Q_{\pi}$  learning. As a policy, MPC planner first uses the infinite-horizon  $Q_{\pi}(s,a_{1:H}) = Q_{\pi}^{H}(s,a_{1:H}) + \gamma^{H}\cdot F_{\pi}(s,a_{1:H})$  to plan and collect rollouts. Then  $F_{\pi}$  is trained with these on policy rollouts while  $Q_{\pi}^{H}$  is also learned like vanilla RaMP via online least square. By incorporating this infinite-horizon variant, our MPC planner can now plan with an infinite horizon during the online phase.
Infinite-horizon Q-function variant is used in all benchmarks in Figure 4.
Figure 5: Finetuning results on MetaWorld and D'Claw. Our method is able to continuously improve by finetuning the Q-basis networks during online training.
B.2 Online Psi Finetuning
While we freeze the Q-basis networks during online training in our setup in Section 2, we can also finetune the Q-basis networks to continuously improve our estimate of the Q-value. After performing reward regression for a number of steps, we can finetune the Q-basis networks on online trajectories by fitting the predicted Q-values to the Monte-Carlo Q values and allowing the gradients to flow through the regression weights. To illustrate the effectiveness of finetuning, we train our algorithm on a set of environments for 12800 steps, where we freeze the regression weights and begin finetuning after 6400 environment steps. As shown in Fig. 5, our method sees a noticeable performance increase with finetuning. We incorporate finetuning in our Hopper and D'Claw experiments, where we finetune the psi network every 800 steps.
B.3 Planning With Uncertainty
Since the Q-basis networks are trained on an offline dataset, they can provide inaccurate Q estimates when online planning goes out of distribution. To encourage the agent to stay within distribution, we train an ensemble of Q-basis networks and augment the online planning objective with a disagreement penalty. Let  $\{\psi_m\}_{m\in [M]}$  denote the ensemble of Q-basis networks. The planning objective becomes
$$
n _ {t} ^ {*} \in \underset {n \in [ N ]} {\operatorname {a r g m a x}} \frac {1}{M} \sum_ {m \in [ M ]} \sum_ {k \in [ K ]} w _ {k} ^ {*} \psi_ {m} \left(s _ {t}, a _ {t: t + H - 1} ^ {n}; v _ {k} ^ {*}\right) - \beta \operatorname {V a r} \left(\left\{\sum_ {k \in [ K ]} w _ {k} ^ {*} \psi_ {m} \left(s _ {t}, a _ {t: t + H - 1} ^ {n}; v _ {k} ^ {*}\right) \right\} _ {m \in [ M ]}\right) \tag {B.1}
$$
where  $\beta$  is a hyperparameter controlling the weight of the penalty. We use  $\beta = 1$  for all our experiments.
B.4 Model-Predictive Path Integral (MPPI)
Our method benefits from more powerful planning algorithms such as model-predictive path integral (MPPI) [53]. MPPI is an sampling-based trajectory optimization method which maintains a distribution of action sequence initialized as isotropic standard Gaussian. In each iteration, MPPI draws  $n$  action sequences from the current distribution and computes their value using the learned Q-basis networks and online regression weights. It then updates the action distribution using a weighted mean and standard deviation of the sampled trajectories, where the weights are computed as the softmax of the values multiplied by a temperature parameter  $\gamma$ . We use an MPPI planner with  $\gamma = 10$  for the D'Claw environment.
C Setup Details and Additional Experiments
In this section, we provide more details of the experiments, including more detailed setup and supplementary results.
C.1 Description of Environments
We describe the details of all used environments such as observation space, action space, reward, offline / online objectives, and dataset collection.
Meta-World All our Meta-World [56] domains share the standard Meta-World observation which includes gripper location, and object locations of all possible objects involved in the Meta-World benchmark. Although the observation space has 39 dimensions, each domain only uses one or two objects so only 7 dimensions are changing any each domain we chose. For pixel observation variants of each domain, we concatenate two  $84 \times 84 \times 3$  RGB images from two views, with a resulting observation dimension of 42336. Each domain has a 4 dimensional action space, corresponding to the delta movement of end-effector in the workspace along with the delta movement of the gripper finger. Metaworld provides a set of 50 goal configurations for each domain. We collect offline dataset following the procedure described in Sec. C.2. The online objective is chosen to be a novel configuration that isn't any of the 50 offline goal configurations. To create the privileged dataset, we choose 25 of the goal configurations as offline objectives. These chosen configurations are the furthest 25 goals from the online objective in the Euclidean distance. We evenly annotate the offline dataset with rewards for each of these goals to form a privileged dataset such that the online objective is out of the distribution of the offline objectives. For different configurations of the same domain, since object locations are in observation and the goal configuration isn't in it, the dynamics is the same. We now describe the objectives of all used meta-world domains, including those used in the appendix.
1. Reach across Wall The objective is to reach a target across a wall. The location of the target is randomized for each goal configuration.
2. Open Door The objective is to open the door to a certain angle by putting the end-effector between the door handle and the door surface. For each configuration, the location of the door is randomized.
3. Turn on Faucet The objective is to turn on a faucet by rotating it along an axis. For each goal configuration, the location of the faucet is randomized.
4. Press Button The objective is to press a button horizontally. The location of the button is randomized for each goal configuration.
Hopper Hopper is an environment with a higher dimensional observation of 11 dimensions and an action space dimension of 3. Hopper is a locomotion environment that requires long-horizon reasoning since a wrong action will make it fall down and touch the ground only after some steps. The objective of the original Hopper environment in OpenAI gym is to train it to run forward. To analyze the performance of CQL and SF on Hopper, we modify the objective to a goal-conditioned variant such that the agent is trained to follow a certain velocity specified by its goal. Similar modifications are common in meta reinforcement learning such as in [12]. We sample a set of 50 goal velocities between 0.5 and 0.75 as offline objectives to collect data in the same way as we did in Metaworld environment. To ensure good coverage of state-action space, we collect data following a set of 50  $\epsilon$ -noised expert policies. For online transfer, we choose four domains with four different online objectives. All the variant domains of Hopper share the same dynamics.
1. Hopper Stand The objective is to stand still and upright. This online objective is on the boundary of the offline objectives' range.
2. Hopper Jump The objective is to jump to a height of 1.5, without moving forward or backward. Such height is typically not achievable when running forward, so this objective is drastically different from offline objectives.
3. Hopper Backward The objective is to run backward at a target speed of 0.5, which is in the opposite direction of the offline objective. Falling down to the ground is penalized.
4. Hopper Sprint The objective is to run forward at a speed of 1.5, which is out of the distribution of offline objectives ranging from 0 to 1.0. The reward function remains the same as that for offline objectives. Only the goal changes.
The 50  $\epsilon$ -noised expert policies used to collect data contains 10 policies from Hopper Stand, 10 policies from Hopper Jump, 10 policies from Hopper Backward, 10 policies from Hopper Sprint and
10 policies from a another hopper environment whose goal is running at a random velocity between 0 and 1. We note the 50 policies doesn't provide any information about their corresponding rewards because the reward is only computed towards the different set of goal between 0.5 and 0.75.
D'Claw D'Claw environment is a dexterous manipulation environment with 24 dimensions of observation space and 9 dimensions of action space. The 9 dimensions correspond to the 9 joints of a robot hand with 3 fingers and 3 joints on each finger. The objective of the environment is to control the hand to rotate a tripod to a certain angle. Angular distance to the target angle is used as the offline objective. To increase the degree of freedom and make the environment more challenging, we allow the tripod to not only rotate but also translate freely on the 2d plane. The initial rotation and position of the tripod are randomized upon every episode. We collect the offline dataset in the same way as in meta-world, training on 50 offline objectives and using  $\epsilon$ -greedy to collect rollouts. At test time we choose a new offline objective angle and annotate the rewards of the privileged dataset in the same way we did for goal conditioned Metaworld environments.
Point Point is an analytical 2D goal-reaching environment with linear dynamics. The reward is defined as the distance to goal minus an action penalty. The offline objectives are negative distances to the randomly selected goals on the 2D plane. The online objectives are novel goals on the plane. Since we are not evaluating CQL and SF on this environment, we don't generate the privileged dataset for Point.
Point Perturbed Point Perturbed shares the same linear dynamics as Point, but features unsafe regions with negative rewards or local maxima with small positive rewards. These perturbations represent out-of-distribution test objectives that cannot be well approximated by a low-dimensional feature vector. Note that the added perturbations make Point Perturbed no longer an instance of the Point environment with a different goal. Instead, it features online objectives that are completely in a different class from Point.
For each domain, we first train 50 policies with SAC [15]. Each policy is trained towards some offline objective of the domain described in Sec. C.1 for 50000 steps. We then use an  $\epsilon$ -greedy version of each trained policy to collect 32000 data points for each domain per offline objective. We choose  $\epsilon = 0.5$ . Such a procedure ensures the dataset has reasonable coverage of the entire state-action space. We note training these policies are fully optional, since RaMP only trajectories without rewards. Datasets collected via intrinsic rewards like curiosity would totally suffice. We also note that the environment used to train expert policies are not necessarily the same as environments used to collect the data points. They should share the same dynamics yet the only the later provides reward information to our offline dataset. In particular, all metaworld data are collected via their original goal conditioned policies while hopper data are collected with a mixture of running and jumping policies in a different goal-velocity-conditioned environment as described in Sec. C.1. We choose the random feature dimension to be 2048. Each dimension in the random feature is extracted by feeding the state-action tuple to a randomly initialized MLP with 2 hidden layers of size of 32. There are therefore 2048 independent, small random MLPs to extract  $\phi$ . All state-action tuples are projected to reward basis  $\phi$  with it.
During offline training phase, we ensemble 8 instances of MLP with 2 hidden layers of size 4096 and train  $\psi$  network following Sec. 3.2. We train  $\psi$  network with a learning rate of  $3\times 10^{-4}$  on the offline dataset for 4 epochs, with a  $\gamma$  decay of 0.9 and batch size 128. We choose the horizon  $H$  to be 16 for Meta-World and Hopper environments and 10 for D'Claw. During online adaptation phase, we first do random exploration for 2500 steps to collect enough data points for linear regression. When doing linear regression, we always concatenate a bias dimension to  $\psi$ . Since online least square makes recomputing  $\omega$  regression fast, we perform an update of weight vector every single step after initial random exploration is finished. In each MPC step, we randomly sample 1024 action sequences. We penalize the predicted rewards with the variance of predictions from all 8 ensembles following Sec. B.3.
Figure 6: Results on Metaworld with high-dimensional pixel observation. RaMP achieves comparable performance to Dreamer on Pixel Reach and Pixel F截图 Open but struggles to perform well on Pixel Door Open. The performance of RaMP on Pixel F截图 Open and Pixel Reach is very similar to its performance in state observation environments.
C.3 Scaling to High-dimensional Pixel Observation
In Sec.4.3, we evaluate RaMP's ability to scale to environments with high dimensional observations. In this section, we go a step further by evaluating RaMP on visual domains, effectively increasing the dimension of the observation space to 42336 as described in Sec. C.1. We use a CNN encoder following the architecture in [32] followed by 2 layer MLP as the random feature network. Both CNN and MLP layers are randomly initialized. Action is projected and concatenated to the first layer of MLP so the random feature would still be conditioned on both action and observation. We compare our method against the Dreamer [16], the state of art model-based learning method optimized for pixel observations. Similar to MBPO, we pre-train dreamer's dynamics branch with offline data before the online phase. As shown in Fig. 6, our method is able to achieve a similar level of performance as Dreamer in two meta-world environments, Pixel Reach and Pixel Faucet Open. RaMP does not see a significant return drop from the variant of the environment with state observation. Given the efficacy of Dreamer, the result still shows RaMP's performance can scale up to pixel observation. However, Dreamer is able to outperform RaMP significantly in an environment like Pixel Door Open. This is likely because random features capture the change in input space rather than reward space. Pixel observations can give random features a lot of noise while important semantic features may not correspond to a big change in pixel space. We note that instead of using random convolution layers, we can use pre-trained encoders to achieve better semantic feature extraction and significantly improve the quality of random features. This is beyond the scope of our work and we leave this for future works.
C.4 Additional Results on Hopper Sprint
Due to page limit, we omitted the plot for Hopper Sprint in Fig.4. Here we provide additional results of RaMP and baselines for it in Fig. 7. The result is consistent with our analysis in the main paper. RaMP matches or outperforms the baselines just as in other Hopper variants. One major difference here is that CQL is performing well for HopperSprint. This is likely because the online objective of Hopper Sprint is at the boundary of the offline objectives as described in Sec. C.1. Given that offline objectives are running at target velocities, the CQL likely learns to not fall down even if the online objective is out of distribution. By not falling down alone, CQL is capable of maintaining a good reward as seen in this case.
C.5 Additional Ablations
Our method builds on the assumption that a linear combination of high dimensional random features can approximate arbitrary reward functions. In the following ablation experiments, we validate this assumption both quantitatively and qualitatively.
Figure 7: Results on Hopper Sprint
Figure 8: Learning curves for different random feature dimensions. Low-dimensional random features suffer from poor convergence performance, whereas high-dimensional random features experience slow adaptation.
Figure 9: Visualization of reward approximated with different numbers of random basis. As the number of random basis increases, the reward approximation becomes more accurate.
Effect of different types of featurization We experiment with three choices of projections: random features parameterized by a deep neural network, random features parameterized by a gaussian matrix, and polynomial features of state and action up to the second order. We evaluate these choices on Point and MetaWorld Reach in Table 2. We see that a linear combination of NN-parametrized random features approximate the true reward well in all tasks. Polynomial features perform well on environments with simple rewards, but struggle as the reward becomes more complex, while Gaussian features are rarely expressive enough.
Effect of random feature dimension
We evaluate our method on Point, Point Perturbed, and Metaworld Reach using  $\{128, 256, 512, 1024, 2048, 4096\}$  random features. We summarize the final returns in Table. 3 and plot the learning curves in Fig. 8. We find that performance degrades as the feature dimension gets lower because the random features are less capable of approximating the true reward. On the other hand, higher-dimensional random features experience slower adaptation. In addition, we provide visualizations of reward approximation in Fig. 9.
Point	Reach
Random	34.0 ± 1.0	820.8 ± 142.0
Gaussian	-3.6 ± 7.4	188.6 ± 49.2
Polynomial	24.5 ± 5.8	162.9 ± 36.4
Table 2: Return for different features. Random features are able to approximate the true reward well across domains. Polynomial features work in simple environments but do not scale to complex rewards. Gaussian features are unable to express the reward.
Scaling with state dimension In Table 4 we evaluate our method on analytical point environments with 2, 3, and 4 state dimensions. All three environments feature distance-to-goal with an action penalty as the reward. We compute the return error stemming from linear regression as well as the Q error stemming from both linear regression and function approximation. We see an increase in both return error and Q error with higher state dimensions, but overall the approximation errors remain reasonably low.
Nonlinear approximation capability In Fig. 10, we visualize the truncated Q value obtained from a linear combination of the random cumulants and compare it to the ground truth Q value approximated by Monte-Carlo sampling. We perform the comparison in Point and Point Perturbed environments.
Table 3: Return as a function of random feature dimension. Low dimensional random features are unable to approximate the true reward with linear regression, leading to degraded convergence performance.
Point	Point Perturbed	Metaworld Reach
128	30.46 ± 1.62	29.61 ± 2.55	765.38 ± 129.98
256	31.53 ± 1.45	31.09 ± 0.04	806.12 ± 104.77
512	33.02 ± 1.10	33.13 ± 2.09	824.56 ± 121.52
1024	33.36 ± 1.10	32.02 ± 2.38	855.14 ± 85.23
2048	34.01 ± 1.04	31.83 ± 1.31	830.84 ± 142.02
4096	33.31 ± 1.15	32.05 ± 1.11	802.45 ± 94.47
Table 4: Approximation error with different state dimensions. As state dimension increases, approximation error increases but remains in a reasonable range.
Point 2D	Point 3D	Point 4D
Return Error	0.42 ± 0.20	1.20 ± 0.61	1.32 ± 0.70
Q Error	0.32 ± 0.05	0.51 ± 0.10	0.61 ± 0.12
Our method provides an accurate estimate of the Q value even in the face of out-of-distribution and highly nonlinear rewards.
In-distribution baselines Finally, we compare the performance of RaMP to baselines on in-distribution online objectives. Our method is designed to make no assumptions about test objectives during policy transfer. As shown in Fig. 4, RaMP outperforms CQL and SF when the online objectives are out of distribution. A natural question to ask is how things will change when the setting satisfies the assumptions of offline-online objective correlation. For example, in a 2D reaching environment, the training dataset may be annotated with either rewards corresponding to only goals on the right half or goals covering the entire plane. When the online objective is to reach a goal on the left half of the plane, it will be out of distribution for the first case while being in distribution for the second. As shown in Fig. 11, when we curate the labeling process of the privileged dataset to satisfy the in-distribution assumption, CQL and SF receive a significant performance boost, while the performance
Figure 10: Visualization of true Q value and approximated Q value. Our method is able to approximate the Q value in the face of out-of-distribution and highly nonlinear rewards.
of our method and MBPO are unaffected as neither algorithm depends on the offline objectives. This serves as a foil to the generalization of our method to out-of-distribution online objectives.
D Detailed Theoretical Results
In this section, we provide the formal statement of the theoretical insights given in §3.4, and corresponding proofs.
D.1 Formal Statement
Theorem D.1. Suppose the offline data in  $\mathcal{D}$  are generated from some distribution  $\rho \in \Delta (\mathcal{S}\times \mathcal{A})$  i.e.,  $(s_h^m,a_h^m)\sim \rho (\cdot ,\cdot)$  and  $s_{h + 1}^{m}\sim \mathcal{T}$  for all  $(m,h)\in [M]\times [H]$ , and  $\inf_{(s,a)\in S\times A}\rho (s,a) = \underline{\rho} >0$  Suppose  $\theta_{k}\sim p(\cdot)$  for all  $k\in [K]$ , and  $\sup_{(s,a,\theta)\in S\times A\times \mathbb{R}^d}|\phi (s,a;\theta)|\leq \kappa$  for some  $\kappa >0$  and
Figure 11: In-distribution results for Successor Features and CQL. Note that RaMP and MBPO are unaffected since they do not depend on the distribution of the offline objectives.