SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\phi (\cdot ,\cdot ;\theta)$  is continuous. For some large enough  $n\coloneqq MH$  , letting  $\lambda = n^{-1 / 2}$  , we have that if  $K = \Omega (\sqrt{n}\log (\kappa^2\sqrt{n} /\delta))$  , then with probability at least  $1 - \delta$  , for any given reward function  $R$
$$
\| \widehat {Q} _ {\pi} (w ^ {*}) - Q _ {\pi} \| _ {\infty} \leq \frac {1}{1 - \gamma} \sqrt {\frac {1}{\rho} \left[ \inf  _ {f \in \mathcal {H}} \underbrace {\sum_ {(s , a) \in \mathcal {S} \times \mathcal {A}} \int (r - f (s , a)) ^ {2} d R _ {s , a} (r) \rho (s , a)} _ {\mathcal {E} (f)} + \mathcal {O} \Big (\frac {\log (1 / \delta)}{\sqrt {n}} \Big) \right]}
$$
for any policy  $\pi$ , where  $\mathcal{H} := \{f = \int \phi(\cdot, \cdot; \theta) w(\theta) dp(\theta) \mid \int |w(\theta)|^2 dp(\theta) < \infty\}$ ,  $w^*$  is the solution to (3.1), and
$$
\widehat {Q} _ {\pi} (s, a; w ^ {*}) := \mathbb {E} \left[ \sum_ {h = 1} ^ {\infty} \gamma^ {h - 1} \sum_ {k \in [ K ]} w _ {k} ^ {*} \phi \left(s _ {h}, a _ {h}; \theta_ {k}\right) \mid s _ {1} = s, a _ {1} = a \right]. \tag {D.1}
$$
The proof of Theorem D.1 can be found in §D.2. It shows that with large enough amount of random features, the  $Q$ -function of any reward function  $R$ , under any policy  $\pi$ , can be approximated accurately up to some inherent error related to the richness of the function class that the features can represent. Note that we here only state the results under some mild and basic assumptions from the random features literature, and are by no means tight. They can be improved in various ways, for example, if the sampling distribution of  $\theta$ ,  $p$ , can be data-dependent, and some stronger assumptions on the data and kernel function classes [41, 29].
Corollary D.2. Suppose the assumptions in Theorem D.1 hold, and additionally the kernel induced function space  $\mathcal{H}$  is rich enough such that  $\inf_{f\in \mathcal{H}}\mathcal{E}(f) = 0$ . Then, with horizon length  $H = \widetilde{\Theta} (\frac{\log(R_{\max} / \epsilon)}{1 - \gamma})$  and  $M = \widetilde{\Theta} (\frac{1}{(1 - \gamma)^{3\epsilon^4}})$  episodes in dataset  $\mathcal{D}$ , and with  $K = \widetilde{\Omega} ((1 - \gamma)^{-2}\epsilon^{-2})$  random features, we have  $\| \widehat{Q}_\pi^H (w^*) - Q_\pi \|_\infty \leq \mathcal{O}(\epsilon)$  for any  $\pi$ , where for each  $(s,a)$ ,  $\widehat{Q}_\pi^H (s,a;w^*)$  is a  $H$ -horizon truncation of (D.1), which can be estimated from the offline dataset  $\mathcal{D}$ .
The proof of Corollary D.2 can be found in §D.2, which specifies the number of random features, the horizon length per episode, and the number of episodes, in order to approximate  $Q_{\pi}$  accurately by using the data in  $\mathcal{D}$ . Note that the number of random features is not excessive and is polynomial in problem parameters. Combining Theorem D.1 and Corollary D.2 leads to the informal statement in Theorem 3.1.
Next, we justify the use of multi-action  $Q$ -functions in planning in a deterministic transition environment, which contains all the environments our empirical results we have evaluated. Recall that for any given reward  $R$ , let  $Q_{\pi}$  denote the  $Q$ -function under policy  $\pi$ . Note that with a slight abuse of notation,  $Q_{\pi}$  can also be the multi-step  $Q$ -function (see definition in §2), and the meaning should be clear from the input, i.e., whether it is  $Q_{\pi}(s,a)$  or  $Q_{\pi}(s,a_1,\dots ,a_H)$ .
Theorem D.3. Let  $\Pi$  be some subclass of Markov stationary policies, i.e., for any  $\pi \in \Pi$ ,  $\pi : S \to \Delta(\mathcal{A})$ . Suppose the transition dynamics  $\mathcal{T}$  is deterministic. For any given reward  $R$ , denote the  $H$ -step policy obtained from  $H$ -step open-loop policy improvement over  $\Pi$  as  $\pi_H': S \to \mathcal{A}^H$ ,
defined as
$$
\pi^{\prime}_{H}(s)\in \operatorname *{argmax}_{(a_{1},\dots ,a_{H})\in \mathcal{A}^{H}}\max_{\pi \in \Pi}Q_{\pi}(s,a_{1},\dots ,a_{H}),
$$
for all  $s \in S$ . Finally, define the value-function under  $\pi_H'$  as  $V_{\pi_H'}(s) \coloneqq Q_{\pi_H'}(s, \pi_H'(s))$ , where  $Q_{\pi_H'}(s, a_{1:H})$  is the fixed-point of the Bellman operator  $\mathcal{T}_{H,\pi_H'}$  defined in (D.8). Then, we have that for all  $s \in S$
$$
V _ {\pi_ {H} ^ {\prime}} (s) \geq \max _ {a _ {1: H}} \max _ {\pi \in \Pi} Q _ {\pi} (s, a _ {1}, \dots , a _ {H}) \geq \max _ {a} \max _ {\pi \in \Pi} Q _ {\pi} (s, a).
$$
D.2 Deferred Proofs
D.2.1 Proof of Theorem D.1
The proof relies on the result of generalization guarantees of learning from random features, with squared loss. Note that one cannot use the results in [40], which dealt with Lipschitz loss function of the form  $c(y', y) = c(y'y)$ . This does not include the squared loss we used in our experiments. Instead, we resort to the results in [41], which also yield a better statistical rate. For the sake of completeness, we re-state the abridged version of one key result therein, Theorem 1 in [41], as follows.
Lemma D.4. Suppose that  $K$  is a kernel with an integral representation  $K(x,x^{\prime}) = \int_{\Omega}\psi (x,w)\psi (x^{\prime},w)dp(w)$ , where  $(\Omega ,p)$  is a probability space and  $\psi :X\times \Omega \to \mathbb{R}$ , where  $X$  is a separable space. Suppose  $\psi$  is continuous and  $|\psi (x,w)|\leq \kappa$  with  $\kappa \in [1, + \infty)$  almost surely, and  $|y|\leq b$  almost surely. Define the expected risk:
$$
\mathcal {E} (f) := \int (f (x) - y) ^ {2} d \rho (x, y),
$$
where  $\rho$  is the distribution where the data samples  $(x_{i},y_{i})_{i = 1}^{n}$ . Define the solution to kernel ridge regression with  $M$  random features as
$$
\widehat {f} _ {\lambda , M} (x) = \phi_ {M} (x) ^ {\top} \widehat {w} _ {\lambda , M}, \quad \text {w i t h} \widehat {w} _ {\lambda , M} := \left(\widehat {S} _ {M} ^ {\top} \widehat {S} _ {M} + \lambda I\right) ^ {- 1} \widehat {S} _ {M} ^ {\top} \widehat {y}, \tag {D.2}
$$
where  $\phi_M(x) \coloneqq \left( \psi(x, w_1), \psi(x, w_2), \dots, \psi(x, w_M) \right) / \sqrt{M}$ ,  $w_i$  are drawn i.i.d. from  $p(\cdot)$ ,  $\widehat{y} \coloneqq (y_1, \dots, y_n) / n^{1/2}$ ,  $\widehat{S}_M^\top \coloneqq \left( \phi_M(x_1), \dots, \phi_M(x_n) \right) / n^{1/2}$ . Then, suppose  $n \geq n_0$ ,  $\lambda = 1/n^{1/2}$  and the number of features  $M \geq c_0 \sqrt{n} \log(108\kappa^2 \sqrt{n}/\delta)$ , we have that with probability at least  $1 - \delta$ ,
$$
\mathcal {E} (\widehat {f} _ {\lambda , M}) - \min _ {f \in \mathcal {H}} \mathcal {E} (f) \leq \frac {c _ {1} \log^ {2} (1 8 / \delta)}{\sqrt {n}},
$$
where  $n_0, c_0, c_1$  are absolute constants,  $\mathcal{H}$  is the reproducing kernel Hilbert space corresponding to the kernel  $K$ .
We then apply Lemma D.4, with  $(x,y)$  in the lemma being replaced by  $\left((s,a),r(s,a)\right)$ ,  $\rho (x,y),p,x,w,M,\lambda$  in the lemma being replaced by  $\rho (s,a)\cdot R_{s,a},p,(s,a),\theta ,K,\lambda$  in our case. Note that Lemma D.4 requires the space  $X$  to be separable, and our finite space  $\mathcal{S}\times \mathcal{A}$  satisfies; it requires  $|y|$  bounded, and our reward is absolutely bounded by  $R_{\mathrm{max}}$ , and thus also satisfies. We hence obtain that with probability at least  $1 - \delta$ , if the number of random features  $K\geq \Omega (\sqrt{n}\log (\sqrt{n}))$  with  $n\coloneqq HM$ , then
$$
\mathbb {E} _ {(s, a) \sim \rho (\cdot , \cdot), r \sim R _ {s, a} (\cdot)} \left(r - \sum_ {k \in [ K ]} w _ {k} ^ {*} \phi (s, a; \theta_ {k})\right) ^ {2} \leq \inf  _ {f \in \mathcal {H}} \mathcal {E} (f) + \mathcal {O} \left(\frac {\log (1 / \delta)}{\sqrt {n}}\right) \tag {D.3}
$$
where we note that  $w^{*} = (w_{1}^{*},\dots ,w_{K}^{*})$  is the solution to (3.1), and the  $\mathcal{E}(f)$  here is defined in Theorem D.1.
For any policy  $\pi$  for the MDP, let  $Q_{\pi}$  denote the  $Q$ -function under policy  $\pi$  and the actual reward function distribution  $R$ , and  $\widehat{Q}_{\pi}(w^{*})$  denote the  $Q$ -function under the estimated reward using random
$$
\widehat {Q} _ {\pi} (s, a; w ^ {*}) := \mathbb {E} \left[ \sum_ {h = 1} ^ {\infty} \gamma^ {h - 1} \widehat {r} (s _ {h}, a _ {h}; w ^ {*}) \Big | s _ {1} = s, a _ {1} = a \right],
$$
$$
\text {w h e r e} \quad \widehat {r} (s, a; w ^ {*}) := \sum_ {k \in [ K ]} w _ {k} ^ {*} \phi (s, a; \theta_ {k}). \tag {D.4}
$$
By Bellman equation, we have that for each  $(s,a)$
$$
\begin{array}{l} \left| Q _ {\pi} (s, a) - \widehat {Q} _ {\pi} (s, a; w ^ {*}) \right| = \left| \int r d R _ {s, a} (r) + \gamma \sum_ {s ^ {\prime}, a ^ {\prime}} Q _ {\pi} (s, a) \mathcal {T} \left(s ^ {\prime} \mid s, a\right) \pi \left(a ^ {\prime} \mid s ^ {\prime}\right) \right. \\ \left. - \widehat {r} (s, a; w ^ {*}) - \gamma \sum_ {s ^ {\prime}, a ^ {\prime}} \widehat {Q} _ {\pi} (s, a; w ^ {*}) \mathcal {T} \left(s ^ {\prime} \mid s, a\right) \pi \left(a ^ {\prime} \mid s ^ {\prime}\right) \right| \\ \leq \left| \int r d R _ {s, a} (r) - \widehat {r} (s, a; w ^ {*}) \right| + \gamma \cdot \left\| Q _ {\pi} - \widehat {Q} _ {\pi} (w ^ {*}) \right\| _ {\infty}. \\ \end{array}
$$
Taking sup over  $s, a$  and organizing the terms, we have
$$
\begin{array}{l} \left\| Q _ {\pi} - \widehat {Q} _ {\pi} \left(w ^ {*}\right) \right\| _ {\infty} \leq \frac {1}{1 - \gamma} \cdot \sup  _ {s, a} \left| \int r d R _ {s, a} (r) - \widehat {r} (s, a; w ^ {*}) \right| \\ \leq \frac {1}{1 - \gamma} \cdot \sqrt {\sum_ {s , a} \left(\int r d R _ {s , a} (r) - \widehat {r} (s , a ; w ^ {*})\right) ^ {2}} (D.5) \\ \leq \frac {1}{1 - \gamma} \cdot \sqrt {\frac {1}{\rho} \sum_ {s , a} \left(\int r d R _ {s , a} (r) - \widehat {r} (s , a ; w ^ {*})\right) ^ {2} \rho (s , a)} (D.6) \\ = \frac {1}{(1 - \gamma) \sqrt {\rho}} \cdot \sqrt {\mathbb {E} _ {(s , a) \sim \rho (s , a)} \left(\int r d R _ {s , a} (r) - \widehat {r} (s , a ; w ^ {*})\right) ^ {2}} (D.7) \\ \end{array}
$$
where (D.5) uses that  $\| \cdot \|_{\infty} \leq \| \cdot \|_2$  for finite-dimensional vectors, (D.6) uses the definition of  $\underline{\rho}$ . Further, by Jensen's inequality, for each  $(s,a)$
$$
\Big (\int r d R _ {s, a} (r) - \widehat {r} (s, a; w ^ {*}) \Big) ^ {2} = \Big (\mathbb {E} _ {r \sim R _ {s, a} (\cdot)} \big [ r - \widehat {r} (s, a; w ^ {*}) \big ] \Big) ^ {2} \leq \mathbb {E} _ {r \sim R _ {s, a} (\cdot)} \big [ r - \widehat {r} (s, a; w ^ {*}) \big ] ^ {2},
$$
which, combined with (D.7) and (D.3), gives that
$$
\begin{array}{l} \left\| Q _ {\pi} - \widehat {Q} _ {\pi} (w ^ {*}) \right\| _ {\infty} \leq \frac {1}{(1 - \gamma) \sqrt {\underline {{\rho}}}} \cdot \sqrt {\mathbb {E} _ {(s , a) \sim \rho (s , a) , r \sim R _ {s , a} (\cdot)} \Big (\int r d R _ {s , a} (r) - \widehat {r} (s , a ; w ^ {*}) \Big) ^ {2}} \\ \leq \frac {1}{(1 - \gamma) \sqrt {\underline {{\rho}}}} \cdot \sqrt {\inf _ {f \in \mathcal {H}} \mathcal {E} (f) + \mathcal {O} \Big (\frac {\log (1 / \delta)}{\sqrt {n}} \Big)}, \\ \end{array}
$$
D.2.2 Proof of Corollary D.2
First, note that with  $H = \Theta\left(\frac{\log(R_{\max} / (\epsilon(1 - \gamma)))}{1 - \gamma}\right)$  ensures that  $\| \widehat{Q}_{\pi}^{H}(w^{*}) - \widehat{Q}_{\pi}(w^{*})\|_{\infty} \leq \mathcal{O}(\epsilon)$ , which can be obtained by the boundedness of  $r(s,a)$  by  $R_{\mathrm{max}}$ , and the fact that
$$
\gamma^ {H} \frac {R _ {\mathrm {m a x}}}{1 - \gamma} = (1 - (1 - \gamma)) ^ {\frac {1}{1 - \gamma} \cdot H (1 - \gamma)} \frac {R _ {\mathrm {m a x}}}{1 - \gamma} \leq \left(\frac {1}{e}\right) ^ {\log (R _ {\mathrm {m a x}} / (\epsilon (1 - \gamma)))} \frac {R _ {\mathrm {m a x}}}{1 - \gamma} = \epsilon .
$$
Furthermore, since Theorem D.1 requires  $n = HM = \widetilde{\Theta} \left( \frac{1}{(1 - \gamma)^4 \epsilon^4} \right)$ , to make sure  $\| \widehat{Q}_{\pi}(w^{*}) - Q_{\pi} \|_{\infty} \leq \epsilon$ . Combining these facts yields the desired result.
D.2.3 Proof of Theorem 3.2
$$
Q _ {H} ^ {\max } (s, a _ {1}, \dots , a _ {H}) := \max  _ {\pi \in \Pi} Q _ {\pi} (s, a _ {1}, \dots , a _ {H}), \text {a n d} Q ^ {\max } (s, a) := \max  _ {\pi \in \Pi} Q _ {\pi} (s, a).
$$
We also define the Bellman operator under the open-loop policy  $\pi_H^\prime$  as follows: for any  $Q\in$ $\mathbb{R}|S|\times |\mathcal{A}^H |$
$$
\mathcal {T} _ {H, \pi_ {H} ^ {\prime}} (Q) (s, a _ {1}, \dots , a _ {H}) = \mathbb {E} \left[ \sum_ {h \in [ H ]} \gamma^ {h - 1} r \left(s _ {h}, a _ {h}\right) + \gamma^ {H} Q \left(s _ {H + 1}, \pi_ {H} ^ {\prime} \left(s _ {H + 1}\right)\right) \mid s _ {1} = s, a _ {1: H} \right]. \tag {D.8}
$$
Note that  $\mathcal{T}_H,\pi_H^{\prime}$  is a contracting operator, and we denote the fixed point of the operator as  $Q_{H,\pi_H'}\in$ $\mathbb{R}^{|S|\times |\mathcal{A}^H|}$ , which is the  $Q$ -value function under open-loop policy  $\pi_H^\prime$ . By definition, we also know that the state-value function under  $\pi_H^\prime$ ,  $V_{H,\pi_H'} = Q_{H,\pi_H'}(s,\pi_H'(s))$ , i.e., by applying the open-loop policy  $\pi_H^\prime$  to the actions  $a_{1:H}$  in  $Q_{H,\pi_H'}(s,a_{1:H})$ .
$$
\begin{array}{l} \mathcal {T} _ {H, \pi_ {H} ^ {\prime}} \left(Q _ {H} ^ {\max }\right) (s, a _ {1}, \dots , a _ {H}) = \mathbb {E} \left[ \sum_ {h \in [ H ]} \gamma^ {h - 1} r \left(s _ {h}, a _ {h}\right) + \gamma^ {H} Q _ {H} ^ {\max } \left(s _ {H + 1}, \pi_ {H} ^ {\prime} \left(s _ {H + 1}\right)\right) \Big | s _ {1} = s, a _ {1: H} \right] \\ = \mathbb {E} \left[ \sum_ {h \in [ H ]} \gamma^ {h - 1} r \left(s _ {h}, a _ {h}\right) + \gamma^ {H} \max  _ {a _ {H + 1: 2 H}} Q _ {H} ^ {\max } \left(s _ {H + 1}, a _ {H + 1: 2 H}\right) \mid s _ {1} = s, a _ {1: H} \right] (D.9) \\ \geq \mathbb {E} \left[ \sum_ {h \in [ H ]} \gamma^ {h - 1} r \left(s _ {h}, a _ {h}\right) + \gamma^ {H} \max  _ {a _ {H + 1: 2 H}} Q _ {\pi} \left(s _ {H + 1}, a _ {H + 1: 2 H}\right) \mid s _ {1} = s, a _ {1: H} \right] (D.10) \\ \geq \mathbb {E} \left[ \sum_ {h \in [ H ]} \gamma^ {h - 1} r \left(s _ {h}, a _ {h}\right) + \gamma^ {H} Q _ {\pi} \left(s _ {H + 1}, \pi \left(s _ {H + 1}\right) \dots , \pi \left(s _ {2 H}\right)\right) | s _ {1} = s, a _ {1: H} \right], (D.11) \\ = Q _ {\pi} \left(s, a _ {1}, \dots , a _ {H}\right), (D.12) \\ \end{array}
$$
for any  $\pi \in \Pi$ , where (D.9) uses the definition of  $\pi_H'$ , (D.10) is due to the definition of  $Q_H^{\mathrm{max}}$ , and (D.11) is by the  $\max_{a_{H + 1:2H}}$ , and (D.12) is by definition. Since (D.12) holds for any  $\pi \in \Pi$ , by the monotonicity of  $\mathcal{T}_{H,\pi_H'}$ , we have
$$
Q _ {H, \pi_ {H} ^ {\prime}} (s, a _ {1: H}) = \lim  _ {k \rightarrow \infty} \left(\mathcal {T} _ {H, \pi_ {H} ^ {\prime}}\right) ^ {k} \left(Q _ {H} ^ {\max }\right) (s, a _ {1: H}) \geq Q _ {H} ^ {\max } (s, a _ {1: H}) \geq \max  _ {\pi \in \Pi} Q _ {\pi} (s, a _ {1: H}). \tag {D.13}
$$
Notice that for all  $s$ , by applying  $\pi_H'(s)$  on both sides of (D.13),
$$
V _ {H, \pi_ {H} ^ {\prime}} (s) = Q _ {H, \pi_ {H} ^ {\prime}} \left(s, \pi_ {H} ^ {\prime} (s)\right) \geq \max  _ {\pi \in \Pi} Q _ {\pi} \left(s, \pi_ {H} ^ {\prime} (s)\right) = \max  _ {a _ {1: H}} \max  _ {\pi \in \Pi} Q _ {\pi} \left(s, a _ {1: H}\right). \tag {D.14}
$$
Further, due to the multi-step maximization, we have
$$
\max_{a_{1:H}}\max_{\pi \in \Pi}Q_{\pi}(s,a_{1},\dots ,a_{H})\geq \max_{a}\max_{\pi \in \Pi}Q_{\pi}(s,a),
$$
which, combined with (D.14), completes the proof.
E Complexity Analysis
The space complexity of RaMP is primarily determined by the number of random features that are needed. As we describe in Corollary D.2, we require  $K = \widetilde{\Omega}((1 - \gamma)^{-2}\epsilon^{-2})$  random features to achieve  $\epsilon$ -order error between the estimated and true Q-function. As the required approximation error decreases, the space needed to store all the features grows sublinearly. Note that this result is for any given reward function (including the target one) under any policy  $\pi$ , but not tied to a specific one (under the realizability assumption of the reward function).
On the other hand, the space complexity of the classical successor feature method is relatively fixed in the above sense: the dimension of the feature is fixed and does not change with the accuracy of approximating the optimal Q-function for the target task. However, the resulting guarantee is also more restricted: it is only for the optimal Q-function of the specific target reward, and also depends on the distance between the previously seen and the target reward functions. Hence, the two space complexities are not necessarily comparable.
F Relationship to Existing Work
We briefly connect our proposed algorithm to prior work.
Successor features for transfer in RL: While successor features [5, 4] have shown the ability to transfer across problems in RL, the two key differences in our framework are (1) using random features rather than requiring learned or pre-provided features and (2) training multi-action Q functions rather than typical  $Q_{\pi}$ . These two changes allow transfer to happen across a broader class of reward functions and not simply be restricted to the policy cover experienced at training time.
Model-based RL: Our work is connected to model based RL in that it disentangles dynamics and rewards, but is crucially different in that it doesn't model one-step evolution of state but rather long term accumulation of random features. This trades off compounding error for generalization error.
Model-free RL: Our work is connected to methods for model-free RL in that it also models a set of Q-functions, but importantly this doesn't correspond to a particular reward, but rather to random features of state. By doing so, we are able to adapt to arbitrary rewards at test-time, rather than being tied to a particular reward function.