SlowGuess/NeurIPS_2023_OCR huggingface dataset:
2We define  $e$  to account for the stochasticity of the initial state distribution.  $e = 0$  when the initial state is deterministic.
Plugging (B.14) back to (B.13), we obtain
$$
\begin{array}{l} \mathbb {E} _ {\overline {{s}} _ {i}} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {i , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} \overline {{s}} _ {i}}{\mathrm {d} \theta} \right\| _ {2} \right] \\ \lesssim 4 L _ {\widehat {f}} \widetilde {L} _ {\pi} \cdot \left(\left\| \frac {\mathrm {d} \widehat {s} _ {i - 1 , n}}{\mathrm {d} \theta} \right\| _ {2} + \left\| \frac {\mathrm {d} \widehat {a} _ {i - 1 , n}}{\mathrm {d} \theta} \right\| _ {2}\right) + L _ {\widehat {f}} \widetilde {L} _ {\pi} \cdot \mathbb {E} _ {\overline {{s}} _ {i - 1}} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {i - 1 , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} \overline {{s}} _ {i - 1}}{\mathrm {d} \theta} \right\| _ {2} \right] + 2 L _ {\widehat {f}} L _ {\theta} \\ \leq 4 L _ {\widehat {f}} \widetilde {L} _ {\pi} \cdot \widehat {K} (i - 1) + L _ {\widehat {f}} \widetilde {L} _ {\pi} \cdot \mathbb {E} _ {\overline {{s}} _ {i - 1}} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {i - 1 , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} \overline {{s}} _ {i - 1}}{\mathrm {d} \theta} \right\| _ {2} \right] + 2 L _ {\widehat {f}} L _ {\theta}, \\ \end{array}
$$
where the last inequality follows from the definition of  $\widehat{K}$  in (B.9).
Applying this recursion gives
$$
\begin{array}{l} \mathbb {E} _ {\overline {{s}} _ {i}} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {i , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} \overline {{s}} _ {i}}{\mathrm {d} \theta} \right\| _ {2} \right] \leq e \big (L _ {\widehat {f}} \widetilde {L} _ {\pi} \big) ^ {i} + \big (4 L _ {\widehat {f}} \widetilde {L} _ {\pi} \cdot \widehat {K} (i - 1) + 2 L _ {\widehat {f}} L _ {\theta} \big) \cdot \sum_ {j = 0} ^ {i - 1} \big (L _ {\widehat {f}} \widetilde {L} _ {\pi} \big) ^ {j} \\ \leq \widetilde {L} _ {\widehat {f}} ^ {i} \widetilde {L} _ {\pi} ^ {i} \Big (e + 4 i \cdot \widetilde {L} _ {\widehat {f}} \widetilde {L} _ {\pi} \cdot \widehat {K} (i - 1) + 2 i \cdot \widetilde {L} _ {\widehat {f}} L _ {\theta} \Big), \\ \end{array}
$$
where the first equality follows from (B.7).
As a consequence, we have from (B.14) that
$$
\mathbb {E} _ {\overline {{a}} _ {i}} \left[ \left\| \frac {\mathrm {d} \widehat {a} _ {i , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} \overline {{a}} _ {i}}{\mathrm {d} \theta} \right\| _ {2} \right] \leq \widetilde {L} _ {\widehat {f}} ^ {i} \widetilde {L} _ {\pi} ^ {i + 1} \Big (e + 4 i \cdot \widetilde {L} _ {\widehat {f}} \widetilde {L} _ {\pi} \cdot \widehat {K} (i - 1) + 2 i \cdot \widetilde {L} _ {\widehat {f}} L _ {\theta} \Big) + 2 L _ {\pi} \widehat {K} (i) + 2 L _ {\theta}.
$$
B.3 Proof of Proposition 5.6
Proof. The analysis of gradient bias differs from that of gradient variance as it involves not only the distribution of approximate states but also the recurrent dependencies of the true value on future timesteps, which must be given extra attention.
In the following analysis, we will first apply similar techniques as those outlined in the previous section to establish an upper bound on the decomposed reward terms in the gradient bias. Afterward, we will address the distribution mismatch issue caused by the recursive structure of  $V^{\pi_{\theta}}$  and the non-recursive structure of the value approximation  $\widehat{V}_{\omega_t}$ .
Step 1: Bound the cumulative reward terms in the gradient bias.
To begin with, we decompose the bias of the reward gradient at timestep  $i \geq 0$  as follows,
$$
\begin{array}{l} \mathbb {E} _ {(s _ {i}, a _ {i}) \sim \mathbb {P} (s _ {i}, a _ {i}), (\widehat {s} _ {i, n}, \widehat {a} _ {i, n}) \sim \mathbb {P} (\widehat {s} _ {i}, \widehat {a} _ {i})} \left[ \left\| \frac {\mathrm {d} r (\widehat {x} _ {i , n})}{\mathrm {d} \theta} - \frac {\mathrm {d} r (x _ {i})}{\mathrm {d} \theta} \right\| _ {2} \right] \\ = \mathbb {E} \left[ \left\| \frac {\mathrm {d} r}{\mathrm {d} \widehat {x} _ {i , n}} \cdot \frac {\mathrm {d} \widehat {x} _ {i , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} r}{\mathrm {d} x _ {i}} \cdot \frac {\mathrm {d} x _ {i}}{\mathrm {d} \theta} \right\| _ {2} \right] \\ \leq \mathbb {E} \left[ \left\| \frac {\mathrm {d} r}{\mathrm {d} \widehat {x} _ {i , n}} \cdot \frac {\mathrm {d} \widehat {x} _ {i , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} r}{\mathrm {d} x _ {i}} \cdot \frac {\mathrm {d} \widehat {x} _ {i , n}}{\mathrm {d} \theta} \right\| _ {2} + \left\| \frac {\mathrm {d} r}{\mathrm {d} x _ {i}} \cdot \frac {\mathrm {d} \widehat {x} _ {i , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} r}{\mathrm {d} x _ {i}} \cdot \frac {\mathrm {d} x _ {i}}{\mathrm {d} \theta} \right\| _ {2} \right] \\ \leq 2 L _ {r} \cdot \widehat {K} (i) + L _ {r} \cdot \left(\mathbb {E} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {i , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} s _ {i}}{\mathrm {d} \theta} \right\| _ {2} \right] + \mathbb {E} \left[ \left\| \frac {\mathrm {d} \widehat {a} _ {i , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} a _ {i}}{\mathrm {d} \theta} \right\| _ {2} \right]\right), \tag {B.15} \\ \end{array}
$$
where  $\mathbb{P}(s_i,a_i)$  and  $\mathbb{P}(\widehat{s}_i,\widehat{a}_i)$  are defined in (4.4) with respect to  $s_0\sim \nu_\pi$ $\widehat{s}_0\sim \nu_{\pi}$
We have from (B.3) that for any  $i \geq 1$ ,
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \frac {\mathrm {d} \widehat {a} _ {i , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} a _ {i}}{\mathrm {d} \theta} \right\| _ {2} \right] \\ = \mathbb {E} \left[ \left\| \frac {\partial \widehat {a} _ {i , n}}{\partial \widehat {s} _ {i , n}} \cdot \frac {\mathrm {d} \widehat {s} _ {i , n}}{\mathrm {d} \theta} + \frac {\partial \widehat {a} _ {i , n}}{\partial \theta} - \frac {\partial a _ {i}}{\partial s _ {i}} \cdot \frac {\mathrm {d} s _ {i}}{\mathrm {d} \theta} - \frac {\partial a _ {i}}{\partial \theta} \right\| _ {2} \right] \\ \end{array}
$$
By the triangle inequality and the Lipschitz assumption, it then follows that
$$
\begin{array}{l} \leq \mathbb {E} \left[ \left\| \frac {\partial \widehat {a} _ {i , n}}{\partial \widehat {s} _ {i , n}} \cdot \frac {\mathrm {d} \widehat {s} _ {i , n}}{\mathrm {d} \theta} - \frac {\partial a _ {i}}{\partial s _ {i}} \cdot \frac {\mathrm {d} \widehat {s} _ {i , n}}{\mathrm {d} \theta} \right\| _ {2} \right] + \mathbb {E} \left[ \left\| \frac {\partial a _ {i}}{\partial s _ {i}} \cdot \frac {\mathrm {d} \widehat {s} _ {i , n}}{\mathrm {d} \theta} - \frac {\partial a _ {i}}{\partial s _ {i}} \cdot \frac {\mathrm {d} s _ {i}}{\mathrm {d} \theta} \right\| _ {2} \right] + \mathbb {E} \left[ \left\| \frac {\partial \widehat {a} _ {i , n}}{\partial \theta} - \frac {\partial a _ {i}}{\partial \theta} \right\| _ {2} \right] \\ \leq 2 L _ {\pi} \cdot \mathbb {E} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {i , n}}{\mathrm {d} \theta} \right\| _ {2} \right] + L _ {\pi} \cdot \mathbb {E} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {i , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} s _ {i}}{\mathrm {d} \theta} \right\| _ {2} \right] + 2 L _ {\theta}. \tag {B.16} \\ \end{array}
$$
Similarly, we have from (B.4) that for any  $i \geq 1$ ,
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {i , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} s _ {i}}{\mathrm {d} \theta} \right\| _ {2} \right] \\ = \mathbb {E} \left[ \left\| \frac {\partial \widehat {s} _ {i , n}}{\partial \widehat {s} _ {i - 1 , n}} \cdot \frac {\mathrm {d} \widehat {s} _ {i - 1 , n}}{\mathrm {d} \theta} + \frac {\partial \widehat {s} _ {i , n}}{\partial \widehat {a} _ {i - 1 , n}} \cdot \frac {\mathrm {d} \widehat {a} _ {i - 1 , n}}{\mathrm {d} \theta} - \frac {\partial s _ {i}}{\partial s _ {i - 1}} \cdot \frac {\mathrm {d} s _ {i - 1}}{\mathrm {d} \theta} - \frac {\partial s _ {i}}{\partial a _ {i - 1}} \cdot \frac {\mathrm {d} a _ {i - 1}}{\mathrm {d} \theta} \right\| _ {2} \right] \\ \end{array}
$$
Applying the triangle inequality to extract the  $\epsilon_{f,t}$  term defined in (4.4), we proceed by
$$
\begin{array}{l} \leq \mathbb {E} \left[ \left\| \frac {\partial \widehat {s} _ {i , n}}{\partial \widehat {s} _ {i - 1 , n}} \cdot \frac {\mathrm {d} \widehat {s} _ {i - 1 , n}}{\mathrm {d} \theta} - \frac {\partial s _ {i}}{\partial s _ {i - 1}} \cdot \frac {\mathrm {d} \widehat {s} _ {i - 1 , n}}{\mathrm {d} \theta} \right\| _ {2} \right] + \mathbb {E} \left[ \left\| \frac {\partial s _ {i}}{\partial s _ {i - 1}} \cdot \frac {\mathrm {d} \widehat {s} _ {i - 1 , n}}{\mathrm {d} \theta} - \frac {\partial s _ {i}}{\partial s _ {i - 1}} \cdot \frac {\mathrm {d} s _ {i - 1}}{\mathrm {d} \theta} \right\| _ {2} \right] \\ + \mathbb {E} \left[ \left\| \frac {\partial \widehat {s} _ {i , n}}{\partial \widehat {a} _ {i - 1 , n}} \cdot \frac {\mathrm {d} \widehat {a} _ {i - 1 , n}}{\mathrm {d} \theta} - \frac {\partial s _ {i}}{\partial a _ {i - 1}} \cdot \frac {\mathrm {d} \widehat {a} _ {i - 1 , n}}{\mathrm {d} \theta} \right\| _ {2} \right] + \mathbb {E} \left[ \left\| \frac {\partial s _ {i}}{\partial a _ {i - 1}} \cdot \frac {\mathrm {d} \widehat {a} _ {i - 1 , n}}{\mathrm {d} \theta} - \frac {\partial s _ {i}}{\partial a _ {i - 1}} \cdot \frac {\mathrm {d} a _ {i - 1}}{\mathrm {d} \theta} \right\| _ {2} \right] \\ \leq \epsilon_ {f, t} \cdot \mathbb {E} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {i - 1 , n}}{\mathrm {d} \theta} \right\| _ {2} + \left\| \frac {\mathrm {d} \widehat {a} _ {i - 1 , n}}{\mathrm {d} \theta} \right\| _ {2} \right] + L _ {f} \cdot \mathbb {E} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {i - 1 , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} s _ {i - 1}}{\mathrm {d} \theta} \right\| _ {2} \right] \\ + L _ {f} \cdot \mathbb {E} \left[ \left\| \frac {\mathrm {d} \widehat {a} _ {i - 1 , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} a _ {i - 1}}{\mathrm {d} \theta} \right\| _ {2} \right] \\ \leq \epsilon_ {f, t} \cdot \widehat {K} (i - 1) + L _ {f} \cdot \mathbb {E} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {i - 1 , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} s _ {i - 1}}{\mathrm {d} \theta} \right\| _ {2} \right] + L _ {f} \cdot \mathbb {E} \left[ \left\| \frac {\mathrm {d} \widehat {a} _ {i - 1 , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} a _ {i - 1}}{\mathrm {d} \theta} \right\| _ {2} \right], \tag {B.17} \\ \end{array}
$$
where the last inequality follows from the definition of  $\hat{K} (i - 1)$  in (B.9).
Combining (B.16) and (B.17), we have
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {i , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} s _ {i}}{\mathrm {d} \theta} \right\| _ {2} \right] \lesssim (\epsilon_ {f, t} + 2 L _ {f} L _ {\pi}) \cdot \widehat {K} (i - 1) + L _ {f} \widetilde {L} _ {\pi} \cdot \mathbb {E} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {i - 1 , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} s _ {i - 1}}{\mathrm {d} \theta} \right\| _ {2} \right] + 2 L _ {f} L _ {\theta} \\ = \left(\left(\epsilon_ {f, t} + 2 L _ {f} L _ {\pi}\right) \cdot \widehat {K} (i - 1) + 2 L _ {f} L _ {\theta}\right) \cdot \sum_ {j = 0} ^ {i - 1} L _ {f} ^ {j} \widetilde {L} _ {\pi} ^ {j} \\ \leq \left(\left(\epsilon_ {f, t} + 2 L _ {f} L _ {\pi}\right) \cdot \widehat {K} (i - 1) + 2 L _ {f} L _ {\theta}\right) \cdot i \cdot \widetilde {L} _ {f} ^ {i} \widetilde {L} _ {\pi} ^ {i}, \tag {B.18} \\ \end{array}
$$
where the last inequality follows from (B.7) and the fact that  $s_0, \widehat{s}_{0,n}$  are sampled from the same initial distribution, and the equality holds by applying the recursion.
Plugging (B.18) into (B.16), we obtain
$$
\mathbb {E} \left[ \left\| \frac {\mathrm {d} \widehat {a} _ {i , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} a _ {i}}{\mathrm {d} \theta} \right\| _ {2} \right] \leq \left[ \left(\epsilon_ {f, t} + 2 L _ {f} L _ {\pi}\right) \cdot \widehat {K} (i - 1) + 2 L _ {f} L _ {\theta} \right] \cdot i \cdot \widetilde {L} _ {f} ^ {i} \widetilde {L} _ {\pi} ^ {i + 1} + 2 L _ {\pi} \widehat {K} (i) + 2 L _ {\theta}. \tag {B.19}
$$
Step 2: Address the state distribution mismatch issue.
The next step is to address the distribution mismatch issue caused by the recursive structure of the value function and the non-recursive structure of the value approximation, i.e., the critic.
We define  $\overline{\sigma}_1(s,a) = \mathbb{P}(s_h = s,a_h = a)$  where  $s_0\sim \nu_\pi, a_i\sim \pi (\cdot |s_i)$ , and  $s_{i + 1}\sim f(\cdot |\bar{s}_i,a_i)$ . In a similar way, we define  $\widehat{\sigma}_{1}(s,a) = \mathbb{P}(\widehat{s}_{h} = s,\widehat{a}_{h} = a)$  where  $\widehat{s}_0\sim \nu_\pi, \widehat{a}_i\sim \pi (\cdot |\widehat{s}_i)$ , and  $\widehat{s}_{i + 1}\sim \widehat{f} (\cdot |\widehat{s}_i,\widehat{a}_i)$ .
Now we are ready to bound the gradient bias. From Lemma B.4, we know that
$$
\begin{array}{l} b _ {t} \leq \kappa \kappa^ {\prime} \cdot \mathbb {E} _ {s _ {0} \sim \nu_ {\pi}, \widehat {s} _ {0, n} \sim \nu_ {\pi}} \left[ \left\| \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (s _ {i}, a _ {i}) - \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (\widehat {s} _ {i, n}, \widehat {a} _ {i, n}) \right\| _ {2} \right] \\ + \kappa^ {\prime} \gamma^ {h} \cdot \mathbb {E} _ {(s _ {h}, a _ {h}) \sim \bar {\sigma} _ {1}, (\widehat {s} _ {h, n}, \widehat {a} _ {h, n}) \sim \widehat {\sigma} _ {1}} \left[ \left\| \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a _ {h}} \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} + \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s _ {h}} \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} - \nabla_ {\theta} \widehat {Q} _ {t} (\widehat {s} _ {h, n}, \widehat {a} _ {h, n}) \right\| _ {2} \right], \tag {B.20} \\ \end{array}
$$
where recall that  $\kappa' = \beta + \kappa \cdot (1 - \beta)$ .
From Lemma B.5, we have for any policy  $\pi_{\theta}$  that the state-action value function is  $L_{Q}$ -Lipschitz continuous, which gives for any  $s \in S$  and  $a \in \mathcal{A}$  that
$$
\left\| \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a} \right\| _ {2} \leq L _ {Q}, \quad \left\| \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s} \right\| _ {2} \leq L _ {Q}. \tag {B.21}
$$
The bias brought by the critic, i.e., the last term on the right-hand side of (B.20), can be further bounded by
$$
\begin{array}{l} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim \bar {\sigma} _ {1}, (\widehat {s} _ {h, n}, \widehat {a} _ {h, n}) \sim \widehat {\sigma} _ {1}} \left[ \left\| \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a _ {h}} \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} + \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s _ {h}} \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} - \nabla_ {\theta} \widehat {Q} _ {t} (\widehat {s} _ {h, n}, \widehat {a} _ {h, n}) \right\| _ {2} \right] \\ = \mathbb {E} _ {\bar {\sigma} _ {1}, \hat {\sigma} _ {1}} \left[ \left\| \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a _ {h}} \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} + \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s _ {h}} \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} - \frac {\partial \widehat {Q} _ {t}}{\partial \widehat {a} _ {h , n}} \cdot \frac {\mathrm {d} \widehat {a} _ {h , n}}{\mathrm {d} \theta} - \frac {\partial \widehat {Q} _ {t}}{\partial \widehat {s} _ {h , n}} \cdot \frac {\mathrm {d} \widehat {s} _ {h , n}}{\mathrm {d} \theta} \right\| _ {2} \right] \\ \leq \mathbb {E} _ {\overline {{\sigma}} _ {1}, \widehat {\sigma} _ {1}} \left[ \left\| \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a _ {h}} \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} - \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a _ {h}} \cdot \frac {\mathrm {d} \widehat {a} _ {h , n}}{\mathrm {d} \theta} \right\| _ {2} + \left\| \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a _ {h}} \cdot \frac {\mathrm {d} \widehat {a} _ {h , n}}{\mathrm {d} \theta} - \frac {\partial \widehat {Q} _ {t}}{\partial \widehat {a} _ {h , n}} \cdot \frac {\mathrm {d} \widehat {a} _ {h , n}}{\mathrm {d} \theta} \right\| _ {2} \right. \\ \left. + \left\| \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s _ {h}} \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} - \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s _ {h}} \cdot \frac {\mathrm {d} \widehat {s} _ {h , n}}{\mathrm {d} \theta} \right\| _ {2} + \left\| \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s _ {h}} \cdot \frac {\mathrm {d} \widehat {s} _ {h , n}}{\mathrm {d} \theta} - \frac {\partial \widehat {Q} _ {t}}{\partial \widehat {s} _ {h , n}} \cdot \frac {\mathrm {d} \widehat {s} _ {h , n}}{\mathrm {d} \theta} \right\| _ {2} \right] \\ \leq L _ {Q} \cdot \left(\mathbb {E} _ {\bar {\sigma} _ {1}, \hat {\sigma} _ {1}} \left[ \left\| \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} - \frac {\mathrm {d} \widehat {a} _ {h , n}}{\mathrm {d} \theta} \right\| _ {2} + \left\| \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} - \frac {\mathrm {d} \widehat {s} _ {h , n}}{\mathrm {d} \theta} \right\| _ {2} \right]\right) + \left(\frac {\gamma^ {h}}{1 - \gamma}\right) ^ {2} \widehat {K} (h) \cdot \epsilon_ {v, t}, \tag {B.22} \\ \end{array}
$$
where the equality follows from the chain rule and the fact that the critic  $\widehat{Q}_t$  has a non-recursive structure, the last inequality follows from (B.21), (B.9) and the definition of  $\epsilon_{v,t}$  in (4.5).
Plugging (B.15) and (B.22) into (B.20), we obtain
$$
\begin{array}{l} b _ {t} \leq \kappa \kappa^ {\prime} \cdot h \cdot \left(L _ {r} \cdot \left(\mathbb {E} _ {\bar {\sigma} _ {1}, \hat {\sigma} _ {1}} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {h , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} \right\| _ {2} \right] + \mathbb {E} _ {\bar {\sigma} _ {1}, \hat {\sigma} _ {1}} \left[ \left\| \frac {\mathrm {d} \widehat {a} _ {h , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} \right\| _ {2} \right]\right) + 2 L _ {r} \cdot \widehat {K} (h)\right) \\ + \kappa^ {\prime} \gamma^ {h} \cdot \left(L _ {Q} \cdot \left(\mathbb {E} _ {\bar {\sigma} _ {1}, \hat {\sigma} _ {1}} \left[ \left\| \frac {\mathrm {d} \widehat {s} _ {h , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} \right\| _ {2} \right] + \mathbb {E} _ {\bar {\sigma} _ {1}, \hat {\sigma} _ {1}} \left[ \left\| \frac {\mathrm {d} \widehat {a} _ {h , n}}{\mathrm {d} \theta} - \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} \right\| _ {2} \right]\right) + \widehat {K} (h) \cdot \left(\frac {\gamma^ {h}}{1 - \gamma}\right) ^ {2} \epsilon_ {v, t}\right), \tag {B.23} \\ \end{array}
$$
Plugging (B.18), (B.19), and (B.9) into the (B.23), we conclude the proof by obtaining
$$
b _ {t} = O \left(\kappa \kappa^ {\prime} h ^ {2} \frac {1 - \gamma^ {h}}{1 - \gamma} \widetilde {L} _ {\hat {f}} ^ {h} \widetilde {L} _ {f} ^ {h} \widetilde {L} _ {\pi} ^ {2 h} \epsilon_ {f, t} + \kappa^ {\prime} h \gamma^ {h} \left(\frac {\gamma^ {h}}{1 - \gamma}\right) ^ {2} \widetilde {L} _ {\hat {f}} ^ {h} \widetilde {L} _ {\pi} ^ {h} \epsilon_ {v, t}\right). \tag {B.24}
$$
Lemma B.3. The expected value gradient over the state distribution  $\mathbb{P}(s_h)$  can be represented by
$$
\mathbb {E} _ {s _ {h} \sim \mathbb {P} (s _ {h})} \left[ \nabla_ {\theta} V ^ {\pi_ {\theta}} (s _ {h}) \right] = \mathbb {E} _ {(s, a) \sim \overline {{\sigma}} _ {1}} \left[ \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a} \cdot \frac {\mathrm {d} a}{\mathrm {d} \theta} + \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s} \cdot \frac {\mathrm {d} s}{\mathrm {d} \theta} \right],
$$
where  $\mathbb{P}(s_h)$  is the state distribution at timestep  $h$  when  $s_0\sim \zeta ,a_i\sim \pi (\cdot |s_i)$ , and  $s_{i + 1}\sim f(\cdot |s_i,a_i)$ .
Proof. At state  $s_h$ , the value gradient can be rewritten as
$$
\begin{array}{l} \nabla_ {\theta} V ^ {\pi_ {\theta}} \left(s _ {h}\right) = \nabla_ {\theta} \mathbb {E} \left[ r \left(s _ {h}, a _ {h}\right) + \gamma \cdot \int_ {S} f \left(s _ {h + 1} \mid s _ {h}, a _ {h}\right) \cdot V ^ {\pi} \left(s _ {h + 1}\right) \mathrm {d} s _ {h + 1} \right] \\ = \nabla_ {\theta} \mathbb {E} \left[ r \left(s _ {h}, a _ {h}\right) \right] + \gamma \cdot \mathbb {E} \left[ \nabla_ {\theta} \int_ {S} f \left(s _ {h + 1} \mid s _ {h}, a _ {h}\right) \cdot V ^ {\pi} \left(s _ {h + 1}\right) \mathrm {d} s _ {h + 1} \right] \\ = \mathbb {E} \left[ \frac {\partial r _ {h}}{\partial a _ {h}} \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} + \frac {\partial r _ {h}}{\partial s _ {h}} \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} \right. \\ \left. + \gamma \int_ {S} \left(\nabla_ {\theta} f \left(s _ {h + 1} \mid s _ {h}, a _ {h}\right) \cdot V ^ {\pi} \left(s _ {h + 1}\right) + f \left(s _ {h + 1} \mid s _ {h}, a _ {h}\right) \cdot \nabla_ {\theta} V ^ {\pi} \left(s _ {h + 1}\right)\right) \mathrm {d} s _ {h + 1} \right] \\ = \mathbb {E} \left[ \frac {\partial r _ {h}}{\partial a _ {h}} \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} + \frac {\partial r _ {h}}{\partial s _ {h}} \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} + \gamma \int_ {\mathcal {S}} \left(\nabla_ {a} f (s _ {h + 1} | s _ {h}, a) \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} \cdot V ^ {\pi} (s _ {h + 1}) \right. \right. \\ \left. + \nabla_ {s} f \left(s _ {h + 1} \mid s _ {h}, a _ {h}\right) \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} \cdot V ^ {\pi} \left(s _ {h + 1}\right) + f \left(s _ {h + 1} \mid s _ {h}, a _ {h}\right) \cdot \nabla_ {\theta} V ^ {\pi} \left(s _ {h + 1}\right)\right) \mathrm {d} s _ {h + 1} \Bigg ], \tag {B.25} \\ \end{array}
$$
where the first equation follows from the Bellman equation and the last two equations hold due to the chain rule. Here, it is worth noting that when  $h \geq 1$ , both  $a_h$  and  $s_h$  have dependencies on all previous timesteps. For any  $h \geq 1$ , we have from the chain rule that  $\nabla_\theta r(s_h, a_h) = \partial r_h / \partial a_h \cdot \mathrm{d}a_h / \mathrm{d}\theta + \partial r_h / \partial s_h \cdot \mathrm{d}s_h / \mathrm{d}\theta$ . This differs from the case when  $h = 0$ , e.g., in the deterministic policy gradient theorem [52], where we can simply write  $\nabla_\theta r(s_h, a_h) = \partial r_h / \partial a_h \cdot \partial a_h / \partial \theta$ .
Rearranging terms in (B.25) gives
$$
\begin{array}{l} \nabla_ {\theta} V ^ {\pi_ {\theta}} \left(s _ {h}\right) = \mathbb {E} \left[ \nabla_ {a} \left(r \left(s _ {h}, a _ {h}\right) + \gamma \int_ {\mathcal {S}} f \left(s _ {h + 1} \mid s _ {h}, a _ {h}\right) \cdot V ^ {\pi} \left(s _ {h + 1}\right) \mathrm {d} s _ {h + 1}\right) \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} \right. \\ + \nabla_ {s} \left(r (s _ {h}, a _ {h}) + \gamma \int_ {\mathcal {S}} f (s _ {h + 1} | s _ {h}, a _ {h}) \cdot V ^ {\pi} (s _ {h + 1}) \mathrm {d} s _ {h + 1}\right) \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} \\ \left. + \gamma \int_ {\mathcal {S}} f \left(s _ {h + 1} \mid s _ {h}, a _ {h}\right) \cdot \nabla_ {\theta} V ^ {\pi} \left(s _ {h + 1}\right) \mathrm {d} s _ {h + 1} \right] \\ = \mathbb {E} \left[ \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a _ {h}} \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} + \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s _ {h}} \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} + \gamma \int_ {\mathcal {S}} f (s _ {h + 1} | s _ {h}, a _ {h}) \cdot \nabla_ {\theta} V ^ {\pi} (s _ {h + 1}) \mathrm {d} s _ {h + 1} \right], \tag {B.26} \\ \end{array}
$$
where the last equation holds since  $Q^{\pi_{\theta}}(s_h, a_h) = r(s_h, a_h) + \gamma \int_{\mathcal{S}} f(s_{h+1} | s_h, a_h) \cdot V^{\pi}(s_{h+1}) \mathrm{d}s_{h+1}$ . By recursively applying (B.26), we obtain
$$
\nabla_ {\theta} V ^ {\pi_ {\theta}} \left(s _ {h}\right) = \mathbb {E} \left[ \int_ {\mathcal {S}} \sum_ {i = h} ^ {\infty} \gamma^ {i - h} \cdot f \left(s _ {i + 1} \mid s _ {i}, a _ {i}\right) \cdot \left(\frac {\partial Q ^ {\pi_ {\theta}}}{\partial a _ {i}} \cdot \frac {\mathrm {d} a _ {i}}{\mathrm {d} \theta} + \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s _ {i}} \cdot \frac {\mathrm {d} s _ {i}}{\mathrm {d} \theta}\right) \mathrm {d} s _ {i + 1} \right]. \tag {B.27}
$$