SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
w ^ {*} = \underset {w} {\operatorname {a r g m i n}} \frac {1}{M H} \sum_ {h \in [ H ], m \in [ M ]} \left(r \left(s _ {h} ^ {m}, a _ {h} ^ {m}\right) - \sum_ {k \in [ K ]} w _ {k} \phi \left(s _ {h} ^ {m}, a _ {h} ^ {m}; \theta_ {k}\right)\right) ^ {2} + \lambda \| w \| _ {2} ^ {2}, \tag {3.1}
$$
where  $\lambda \geq 0$  is the regularization coefficient, and  $r(s_h^m,a_h^m)\sim R_{s_h^m,a_h^m}$ . Due to the use of random features, Eq. (3.1) is a ridge regression problem, and can be solved efficiently.
Given these weights, it is easy to estimate an approximate multi-step Q-function for the true reward on the new task by linearly combining the Q-basis functions learned in the offline training phase  $\{\psi(\cdot, \cdot; \nu_k^*)\}_{k \in [K]}$  according to the same coefficient vector  $w^*$ . This follows from the additive nature of reward and linearity of expectation. In particular, if the reward function  $r(s, a) = \sum_{k \in [K]} w_k^* \phi(s, a; \theta_k)$  holds approximately, which will be the case for large enough  $K$  and rich enough  $\phi$ , then the approximate Q-function for the true test-time reward under the sequence  $\{a_1, \dots, a_H\}$  satisfies  $Q^H(s_1, a_{1:H}) := \mathbb{E}_{s_{h+1} \sim \mathcal{T}(s_h, a_h)} \left[ \sum_{h \in [H]} \gamma^{h-1} R_{s_h, a_h} \right] \approx \sum_{k \in [K]} w_k^* \psi(s_1, a_{1:H}; \nu_k^*)$ , where  $\{w_k^*\}_{k \in [K]}$  is the solution to the regression problem (Eq. (3.1)) and  $\{\nu_k^*\}_{k \in [K]}$  is the solution to the Q-basis fitting problem described in Section 3.2.
3.3.2 Planning with Model Predictive Control
To obtain the optimal sequence of actions, we can use the inferred approximate Q-function for the true reward  $Q^{H}(s_{1},a_{1:H})$  for online planning at each time  $t$  in the new task: at state  $s_t$ , we conduct standard model-predictive control techniques with random shooting [35, 52], i.e., randomly generating  $N$  sequences of actions  $\{a_1^n,\dots ,a_H^n\}_{n\in [N]}$ , and find the action sequence with the maximum Q-value such that  $n_t^* \in \mathrm{argmax}_{n\in [N]}\sum_{k\in [K]}w_k^*\psi (s_t,a_{t:t + H - 1}^n;\nu_k^*)$ .
We note that this finite-horizon planning is an approximation of the actual infinite-horizon planning, which may be enhanced by an additional reward-to-go term in the objective. We provide such an enhancement with details in Appendix B.1, although we found minimal effects on empirical performance. We then execute  $a_{t}^{n_{t}^{*}}$  from the sequence  $n_t^*$ , observe the new state  $s_{t + 1}$ , and replan. We refer readers to Appendix F for a detailed connection of our proposed method to existing work and Appendix A for pseudocode.
3.4 Theoretical Justifications
We now provide some theoretical justifications for the methodology we adopt. To avoid unnecessary nomenclature of measures and norms in infinite dimensions, we in this section consider the case that  $S$  and  $\mathcal{A}$  are discrete (but can be enormously large). Due to space limitation, we present an abridged version of the results below, and defer the detailed versions and proofs in Appendix D. We first state the following result on the expressiveness of random cumulants.
Theorem 3.1 ( $Q$ -function approximation; Informal). Under standard coverage and sampling assumptions of offline dataset  $\mathcal{D}$ , and standard assumptions on the boundedness and continuity of random features  $\phi(s, a; \theta)$ , it follows that with horizon length  $H = \widetilde{\Theta} \left( \frac{\log(R_{\max} / \epsilon)}{1 - \gamma} \right)$  and  $M = \widetilde{\Theta} \left( \frac{1}{(1 - \gamma)^3 \epsilon^4} \right)$  episodes in dataset  $\mathcal{D}$ , and with  $K = \widetilde{\Omega} \left( (1 - \gamma)^{-2} \epsilon^{-2} \right)$  random features, we have that for any given reward function  $R$ , and any policy  $\pi$
$$
\begin{array}{c} \| \widehat {Q} _ {\pi} ^ {H} (w ^ {*}) - Q _ {\pi} \| _ {\infty} \leq \mathcal {O} (\epsilon) + \mathcal {O} \Big (\sqrt {\inf  _ {f \in \mathcal {H}} \mathcal {E} (f)} \Big) \\ \widehat {x} _ {v} \end{array}
$$
with high probability, where  $\widehat{Q}_{\pi}^{H}(s,a;w^{*})$  is defined as  $\mathbb{E}\left[\sum_{h=1}^{H}\gamma^{h-1}\sum_{k\in [K]}w_k^*\phi(s_h,a_h;\theta_k)\mid s_1 = s,a_1 = a\right]$  for each  $(s,a)$  and can be estimated from the offline dataset  $\mathcal{D}$ ;  $\inf_{f\in \mathcal{H}}\mathcal{E}(f)$  is the infimum expected risk over the function class  $\mathcal{H}$  induced by  $\phi$ .
Theorem 3.1 is an informal statement of the results in Appendix D.2, which specifies the number of random features, the horizon length per episode, and the number of episodes, in order to approximate  $Q_{\pi}$  accurately by using the data in  $\mathcal{D}$ , under any given reward function  $R$  and policy  $\pi$ . Note that the number of random features is not excessive and is polynomial in problem parameters. We also note that the results can be improved under stronger assumptions of the sampling distributions  $p$  and kernel function classes [41, 29].
Next, we justify the use of multi-step  $Q$ -functions in planning in a deterministic transition environment, which contains all the environments our empirical results we will evaluate later. Recall that for any given reward  $R$ , let  $Q_{\pi}$  denote the  $Q$ -function under policy  $\pi$ . Note that with a slight abuse of notation,  $Q_{\pi}$  can also be the multi-step  $Q$ -function (see definition in Section 2), and the meaning should be clear from the input, i.e., whether it is  $Q_{\pi}(s,a)$  or  $Q_{\pi}(s,a_1,\dots ,a_H)$ .
Theorem 3.2. Let  $\Pi$  be some subclass of Markov stationary policies, i.e., for any  $\pi \in \Pi$ ,  $\pi : S \to \Delta(\mathcal{A})$ . Suppose the transition dynamics  $\mathcal{T}$  is deterministic. For any given reward  $R$ , denote the  $H$ -step policy obtained from  $H$ -step policy improvement over  $\Pi$  as  $\pi_H': S \to \mathcal{A}^H$ , defined as  $\pi_H'(s) \in \operatorname{argmax}_{(a_1, \dots, a_H) \in \mathcal{A}^H} \max_{\pi \in \Pi} Q_\pi(s, a_1, \dots, a_H)$ , for all  $s \in S$ . Let  $V_{\pi_H'}$  denote the value function under the policy  $\pi_H'$  (see formal definition in Appendix D.1). Then, we have that for all  $s \in S$
$$
V _ {\pi_ {H} ^ {\prime}} (s) \geq \max  _ {a _ {1: H}} \max  _ {\pi \in \Pi} Q _ {\pi} (s, a _ {1}, \dots , a _ {H}) \geq \max  _ {a} \max  _ {\pi \in \Pi} Q _ {\pi} (s, a).
$$
The proof of Theorem 3.2 can be found in Appendix D.2. The result can be viewed as a generalization of the generalized policy iteration result in [5] to multi-step action-sequence policies. Taking  $\Pi$  to be the set of policies that generate the data, the result shows that the value function of the greedy multi-action policy improves over all the possible  $H$ -step multi-action policies, with the policy after step  $H$  to be any policy in  $\Pi$ . Moreover, the value function by  $\pi_H^{\prime}$  also improves overall one-step policies if the policy after the first step onwards follows any policy in  $\Pi$ . This is due to the fact that  $\Pi$  (coming from data) might be a much more restricted policy class than any action sequence  $a_{1:H}$ .
In this section, we aim to answer the following research questions: (1) Does RaMP allow for effective transfer of behaviors across tasks with varying rewards? (2) Does RaMP scale to domains with high-dimensional observation and action spaces and long horizons? (3) Which design decisions in RaMP enable better transfer?
Across several domains, we evaluate the ability of RaMP to leverage the knowledge of shared dynamics from an offline dataset to quickly solve new tasks with arbitrary rewards.
Figure 3: We evaluate our method on manipulation, locomotion, and high dimensional action space environments. The green arrow in each environment indicates the online objective for policy transfer while the red arrows are offline objectives used to label rewards for the privileged dataset.
Figure 4: Reward transfer results on Metaworld, Hopper and D'Claw environments. RaMP adapts to novel rewards more rapidly than MBPO, Successor Features, and CQL baselines.
Offline Dataset Construction: For each domain, we have an offline dataset collected by a behavior policy as described in Appendix C.2. Typically this behavior policy is a mixture of noisy policies accomplishing different objectives in each domain. Although RaMP and other model-based methods do not require knowledge of any reward from the offline dataset, other baseline comparisons will require privileged information. Baseline comparison methods like model-free RL and successor features require the provision of a set of training objectives, as well as rewards labeled for these objectives on state actions from the offline dataset. We call such objectives "offline objectives" and a dataset annotated with these offline objectives and rewards a privileged dataset.
Test-time adaptation: At test-time, we select a novel reward function for online adaptation, referred to as 'online objective' below. The online objective may correspond to rewards conditioned on different goals or even arbitrary rewards, depending on the domain. Importantly, the online objective need not be drawn from the same distribution as the privileged offline objectives.
Given this problem setup, we compare RaMP with a variety of baselines. (1) MBPO [19] is a model-based RL method that learns a standard one-step dynamics model and uses actor-critic methods to plan in the model. We pre-train the dynamics model for MBPO on the offline dataset. (2) PETS [8] is a model-based RL method that learns an ensemble of one-step dynamics models and performs MPC. We pre-train the dynamics model on the offline dataset and use the cross-entropy method (CEM) to plan. (3) Successor feature (SF) [5] is a framework for transfer learning in RL as described in Sec. 1.1. SF typically assumes access to a set of policies towards different goals along with a learned featurization, so we provide it with the privileged dataset to learn a set of training policies. We also learn successor features with the privileged dataset. (4) CQL [24]: As an oracle comparison, we compare with a goal-conditioned variant of an offline RL algorithm (CQL). CQL is a model-free offline RL algorithm that learns policies from offline data. While model-free offline RL naturally struggles to adapt to arbitrarily changing rewards, we provide CQL with information about the goal at both training and testing time. CQL is trained on the distribution of training goals on the offline dataset, and finetuned on the new goals at test time. The CQL comparison is assuming access to more information than RaMP. Each method is benchmarked on each domain with 4 seeds.
4.2 Transfer to Unseen Rewards
We first evaluate the ability of RaMP to learn from an offline dataset and quickly adapt to unseen test rewards in 4 robotic manipulation environments from Meta-World [56]. We consider skills like
reaching a target across the wall, opening a door, turning on a faucet, and pressing a button while avoiding obstacles, which are challenging for typical model-based RL algorithms (Fig. 3). Each domain features 50 different possible goal configurations, each associated with a different reward but the same dynamics. The privileged offline objectives consist of 25 goal configurations. The test-time reward functions are drawn from the remaining 25 "out-of-distribution" reward functions. We refer the reader to Appendix C.1 for details.
As shown in Fig 4, our method adapts to test reward most quickly across all four meta-world domains. MBPO slowly catches up with our performance with more samples, since it still needs to learn the Q function from scratch even with the dynamics branch trained. PETS adapts faster than MBPO but performs worse than our method as it suffers from compounding error. In multiple environments, successor features barely transfer to the online objective as it entangles policies that are not close to those needed for the online objective. Goal-conditioned CQL performs poorly in all tasks as it has a hard time generalizing to out-of-distribution goals. In comparison, RaMP is able to deal with arbitrary sets of test time rewards, since it does not depend on the reward distribution at training time.
4.3 Scaling to Tasks with Longer Horizons and High Dimensions
We further evaluate the ability of our method to scale to tasks with longer horizons. We consider locomotion domains such as the Hopper environment from OpenAI Gym [7]. We chose the offline objectives to be a mixture of skills such as standing, sprinting, jumping, or running backward with goals defined as target velocities or heights. The online objectives consist of unseen goal velocities or heights for standing, sprinting, jumping or running. As shown in Fig. 4, our method maintains the highest performance when adapting to novel online objectives, as it avoids compounding errors by directly modeling accumulated random features. MBPO adapts at a slower rate since higher dimensional observation and longer horizons increase the compounding error of model-based methods. We note that SF is performing reasonably well, likely because the method also reduces the compounding error compared to MBPO, and it has privileged information.
To understand whether RaMP can scale to higher dimensional state-action spaces, we consider a dexterous manipulation domain (referred to as the D'Claw domain in Fig 3). This domain has a 9 DoF action space controlling each of the joints of the hand as well as a 16-dimensional state space including object position. The offline dataset is collected moving the object to different orientations, and the test-time rewards are tasked with moving the object to new orientations (as described in Appendix C.1). Fig 4 shows that both Q-estimation and planning with model-predictive control remain effective when action space is large. In Appendix C.3, we test our method on environments with even higher dimensional observations such as images.
4.4 Ablation of Design Choices
To understand what design decisions in RaMP enable better transfer and scaling, we conduct ablation studies on various domains, including an analytical 2D point goal-reaching environment and its variants (described in Appendix C.1), as well as the classic double pendulum environment and meta-world reaching. We report an extensive set of ablations in Appendix C.5.
Reduction of compounding error with multi-step Q functions
We hypothesize that our method does not suffer from compounding errors in the same way that feedforward dynamics models do.
In Table 1, we compare the approximation error of truncated Q values computed with (1) multi-step Q functions obtained as a linear combination of random cumulants (Ours), and (2) rollouts of a dynamics model (MBRL). We train the methods on the offline data and evaluate on the data from a novel task at test time. As shown in Table 1, our method outperforms feedforward dynamics models.
Hopper	Pendulum
Ours	25.7 ± 5.4	65.1 ± 0.4
MBRL	50.2 ± 9.7	395.4 ± 5.8
Table 1: Policy evaluation error. Feedforward dynamics model suffers from compounding error that is particularly noticeable in domains with high action dimensions or chaotic dynamics. Our method achieves low approximation error in both domains.
In this work, we introduce RaMP, a method that leverages diverse unlabeled offline data to learn models of long horizon dynamics behavior, while being able to naturally transfer across tasks with different reward functions. To this end, we propose to learn the long-term evolution of random features
under different action sequences, which can be generated in a self-supervised fashion without reward labels. This way, we are able to disentangle dynamics, rewards, and policies, while without explicitly learning the dynamic model. We validate our approach across a number of simulated robotics and control domains, with superior transfer ability than baseline comparisons. There are also limitations of the current approach we would like to address in the future work: the current Monte-Carlo-based value function estimation may suffer from high variance, and we hope to incorporate the actor-critic algorithms [23] in RaMP to mitigate it; current approach is only evaluated on simulated environments, and we hope to further test it on real robotic platforms.
We would like to thank Russ Tedrake, Max Simchowitz, Anurag Ajay, Marius Memmel, Terry Suh, Xavier Puig, Sergey Levine, Seohong Park and many other members of the Improbable AI lab at MIT and the WEIRD Lab at University of Washington for valuable feedback and discussions.
[1] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. P. Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pages 5048–5058, 2017.
[2] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. CoRR, abs/1707.01495, 2017.
[3] K. Asadi, D. Misra, and M. L. Littman. Lipschitz continuity in model-based reinforcement learning. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 264-273. PMLR, 2018.
[4] A. Barreto, D. Borsa, J. Quan, T. Schaul, D. Silver, M. Hessel, D. Mankowitz, A. Zidek, and R. Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pages 501-510. PMLR, 2018.
[5] A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. P. van Hasselt, and D. Silver. Successor features for transfer in reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
[6] E. Bingham and H. Mannila. Random projection in dimensionality reduction: applications to image and text data. In Knowledge Discovery and Data Mining, 2001.
[7] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. CoRR, abs/1606.01540, 2016.
[8] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. CoRR, abs/1805.12114, 2018.
[9] K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman. Quantifying generalization in reinforcement learning. CoRR, abs/1812.02341, 2018.
[10] M. P. Deisenroth and C. E. Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In L. Getoor and T. Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July 2, 2011, pages 465-472. Omnipress, 2011.
[11] A. Filos, C. Lyle, Y. Gal, S. Levine, N. Jaques, and G. Farquhar. Psiphi-learning: Reinforcement learning with demonstrations using successor features and inverse temporal difference learning. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 3305-3317. PMLR, 2021.
[12] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks, 2017.
Supplementary Materials for "Self-Supervised Reinforcement Learning that Transfers using Random Features"
Algorithm 1 Model-Free Transfer with Randomized Cumulants and Model-Predictive Control
1: Input: Offline dataset  $\mathcal{D}$  given in Section 2, distribution  $p$  over  $\mathbb{R}^d$ , number of random features  $K$
2: Offline Training Phase:
3: Randomly sample  $\{\theta_k\}_{k\in [K]}$  with  $\theta_{k}\sim p$  , and construct dataset
$$
\mathcal {D} _ {\phi} = \left\{\left(\left(s _ {1} ^ {m}, a _ {1: H} ^ {m}\right), \sum_ {h \in [ H ]} \gamma^ {h - 1} \phi \left(s _ {h} ^ {m}, a _ {h} ^ {m}; \theta_ {k}\right)\right) \right\} _ {m \in [ M ], k \in [ K ]}.
$$
4: Fit random Q-basis functions  $\psi (\cdot ,\cdot ,\nu_k):\mathcal{S}\times \mathcal{A}^H\to \mathbb{R}$  for  $k\in [K]$  by minimizing the loss over the dataset  $\mathcal{D}_{\phi}$ ,
$$
\left\{\nu_ {k} ^ {*} \right\} _ {k \in [ K ]} \in \underset {\{\nu_ {k} \} _ {k \in [ K ]}} {\operatorname {a r g m i n}} \frac {1}{M} \sum_ {m \in [ M ], k \in [ K ]} \left(\psi (s _ {1} ^ {m}, a _ {1: H} ^ {m}; \nu_ {k}) - \sum_ {h \in [ H ]} \gamma^ {h - 1} \phi (s _ {h} ^ {m}, a _ {h} ^ {m}; \theta_ {k})\right) ^ {2}.
$$
5: Online Planning Phase:
6: Fit the testing task's reward function  $r(\cdot, \cdot)$  with linear regression on random features:
$$
w ^ {*} \in \operatorname * {a r g m i n} _ {w} \frac {1}{M H} \sum_ {h \in [ H ], m \in [ M ]} \left(r (s _ {h} ^ {m}, a _ {h} ^ {m}) - \sum_ {k \in [ K ]} w _ {k} \phi (s _ {h} ^ {m}, a _ {h} ^ {m}; \theta_ {k})\right) ^ {2} + \lambda \| w \| _ {2} ^ {2}
$$
where  $r(s_h^m, a_h^m) \sim R_{s_h^m, a_h^m}$ .
7: Sample  $s_1 \sim \mu_0$
8: for time index  $t = 1, \dots$  do
9: Randomly generate  $N$  sequences of actions  $\{a_1^n,\dots ,a_H^n\}_{n\in [N]}$
10: Find the best sequence such that
$$
n _ {t} ^ {*} \in \operatorname * {a r g m a x} _ {n \in [ N ]} \sum_ {k \in [ K ]} w _ {k} ^ {*} \psi (s _ {t}, a _ {t: t + H - 1} ^ {n}; v _ {k} ^ {*}).
$$
Execute  $a_{t}^{n_{t}^{*}}$  from the sequence  $n_t^*$ , observe the new state  $s_{t + 1}\sim \mathcal{T}(s_t,a_t^{n_t^*})$  11: end for
Our implementation contains a few optional components that are omitted from main paper due to page limit. We note that RaMP's performance already surpasses the baselines even without them. These components are introduced for the completeness of our main algorithm and leads to better sample efficiency in different situations.
B.1 Infinite-Horizon  $Q$ -function Variant
While our setup in Section 2 uses a finite-horizon  $Q_{\pi}^{H}$  to approximate  $Q_{\pi}$ , our method can also plan with an infinite-horizon  $Q$  function during the online phase. In this section, we describe one compatible way to learn an infinite-horizon  $Q$ -function while still enjoying the benefits of RaMP in the case with deterministic transition dynamics. We find this variant leads to continual improvement and thus better performance in a few longer-horizon environments.
We first notice that an infinite-horizon  $Q$ -function  $Q_{\pi}$  can be decomposed into the discounted sum of an  $H$ -step reward and a discounted value function under policy  $\pi$  evaluated at  $s_{t + H}$ :
$$
Q_{\pi}(s^{\prime},a^{\prime}) = \mathbb{E}_{\substack{a_{t}\sim \pi (\cdot \mid s_{t})\\ s_{t + 1}\sim \mathcal{T}(\cdot \mid s_{t},a_{t})}}\Bigl[\gamma^{H}V_{\pi}(s_{H + 1}) + \sum_{t = 1}^{H}\gamma^{t - 1}r(s_{t},a_{t})\big|  s_{1} = s^{\prime},a_{1} = a^{\prime}\Bigr]
$$
where  $V_{\pi}(s^{\prime}) = \mathbb{E}_{a_t\sim \pi (\cdot |s_t)}\Big[\sum_{t = 1}^{\infty}\gamma^{t - 1}r(s_t,a_t)\Big|s_1 = s^{\prime}\Big]$
Given a policy  $\pi$ , value function  $V_{\pi}^{\theta}$  parameterized by  $\theta$  can be learned via gradient descent and Monte-Carlo method:
$$
\theta \leftarrow \theta - \alpha \nabla_ {\theta} \left| \left| V _ {\pi} ^ {\theta} \left(s _ {t}\right) - \left(r _ {t} + \gamma V _ {\pi} ^ {\theta^ {\prime}} \left(s _ {t + 1}\right)\right) \right| \right| _ {2} ^ {2}, \text {f o r s a m p l e d} \left(s _ {t: t + 1}, a _ {t}, r _ {t}\right) \sim \tau_ {\pi}
$$
where  $\tau_{\pi}$  is trajectory rollouts collected with current policy  $\pi$  and  $\theta'$  is a target network that gets updated by  $\theta$  with momentum.
Now consider our multi-step setup. Our multi-step  $Q$ -function can also be written as the sum of our  $H$  step approximation and discounted value function at  $s_{H + 1}$ :
$$
\begin{array}{l} Q_{\pi}(s,\widetilde{a}_{1:H}) = \mathbb{E}_{\substack{a_{H + t}\sim \pi (\cdot \mid s_{H + t})\\ s_{t + 1}\sim \mathcal{T}(\cdot \mid s_{t},a_{t})}}\Big[\sum_{t = 1}^{\infty}\gamma^{t - 1}r_{t}\mid s_{1} = s,a_{1} = \widetilde{a}_{1},\dots ,a_{H} = \widetilde{a}_{H}\Big] \\ = Q _ {\pi} ^ {H} \left(s, \widetilde {a} _ {1: H}\right) + \gamma^ {H} V _ {\pi} \left(s _ {H + 1}\right), \\ \end{array}
$$