SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $\overline{\sigma}_2(s,a) = (1 - \gamma)\cdot \sum_{i = h}^{\infty}\gamma^{i - h}\cdot \mathbb{P}(s_i = s,a_i = a)$ , where  $s_0\sim \zeta$ ,  $a_i\sim \pi (\cdot |s_i)$ , and  $s_{i + 1}\sim f(\cdot |s_i,a_i)$ . By definition we have
$$
\begin{array}{l} \sigma (s, a) = (1 - \gamma) \cdot \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot \mathbb {P} \left(s _ {i} = s, a _ {i} = a\right) + \gamma^ {h} \cdot \overline {{\sigma}} _ {1} (s, a) \\ = (1 - \gamma) \cdot \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot \mathbb {P} \left(s _ {i} = s, a _ {i} = a\right) + \gamma^ {h} \cdot \bar {\sigma} _ {2} (s, a). \\ \end{array}
$$
Therefore we have the equivalence  $\overline{\sigma}_1(s,a) = \overline{\sigma}_2(s,a)$ .
By taking the expectation over  $s_h$  in (B.27), we have the stated result, i.e.,
$$
\mathbb {E} _ {s _ {h} \sim \mathbb {P} (s _ {h})} \big [ \nabla_ {\theta} V ^ {\pi_ {\theta}} (s _ {h}) \big ] = \mathbb {E} _ {(s, a) \sim \overline {{\sigma}}} _ {2} \left[ \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a} \cdot \frac {\mathrm {d} a}{\mathrm {d} \theta} + \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s} \cdot \frac {\mathrm {d} s}{\mathrm {d} \theta} \right] = \mathbb {E} _ {(s, a) \sim \overline {{\sigma}}} _ {1} \left[ \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a} \cdot \frac {\mathrm {d} a}{\mathrm {d} \theta} + \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s} \cdot \frac {\mathrm {d} s}{\mathrm {d} \theta} \right].
$$
Lemma B.4. Recall that the state distribution  $\mu_{\pi}$  where  $\widehat{s}_{0,n}$  is sampled from is of the form  $\mu_{\pi}(s) = \beta \cdot \nu_{\pi}(s) + (1 - \beta)\cdot \zeta (s)$ . The gradient bias  $b_{t}$  at any iteration  $t$  satisfies
$$
\begin{array}{l} b _ {t} \leq \kappa (\beta + \kappa \cdot (1 - \beta)) \cdot \mathbb {E} _ {s _ {0} \sim \nu_ {\pi}, \widehat {s} _ {0, n} \sim \nu_ {\pi}} \left[ \left\| \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (s _ {i}, a _ {i}) - \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (\widehat {s} _ {i, n}, \widehat {a} _ {i, n}) \right\| _ {2} \right] \\ + (\beta + \kappa \cdot (1 - \beta)) \gamma^ {h}. \\ \end{array}
$$
$$
\mathbb {E} _ {(s _ {h}, a _ {h}) \sim \overline {{\sigma}} _ {1}, (\widehat {s} _ {h, n}, \widehat {a} _ {h, n}) \sim \widehat {\sigma} _ {1}} \left[ \left\| \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a _ {h}} \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} + \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s _ {h}} \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} - \nabla_ {\theta} \widehat {Q} _ {t} (\widehat {s} _ {h, n}, \widehat {a} _ {h, n}) \right\| _ {2} \right].
$$
Proof. To begin, we decompose the gradient bias by
$$
\begin{array}{l} b _ {t} = \left\| \nabla_ {\theta} J \left(\pi_ {\theta_ {t}}\right) - \mathbb {E} \left[ \widehat {\nabla} _ {\theta} J \left(\pi_ {\theta_ {t}}\right) \right] \right\| _ {2} \\ = \left\| \mathbb {E} \left[ \nabla_ {\theta} J \left(\pi_ {\theta_ {t}}\right) - \widehat {\nabla} _ {\theta} J \left(\pi_ {\theta_ {t}}\right) \right] \right\| _ {2} \tag {B.28} \\ = \left\| \mathbb {E} _ {s _ {0} \sim \zeta , \widehat {s} _ {0, n} \sim \mu_ {\pi}} \left[ \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r \left(s _ {i}, a _ {i}\right) + \gamma^ {h} \cdot \nabla_ {\theta} V ^ {\pi_ {\theta}} \left(s _ {h}\right) - \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r \left(\widehat {s} _ {i, n}, \widehat {a} _ {i, n}\right) \right. \right. \\ \left. \left. - \gamma^ {h} \cdot \nabla_ {\theta} \widehat {V} _ {t} (\widehat {s} _ {h, n}) \right] \right\| _ {2}, \\ \end{array}
$$
where we note that  $s_0$  and  $\widehat{s}_{0,n}$  are sampled from  $\zeta$  and  $\mu_{\pi}$  following the definition of the RL objective and the form of gradient estimator, respectively.
For  $\mu_{\pi}(s) = \beta \cdot \nu_{\pi}(s) + (1 - \beta)\cdot \zeta (s)$ , let  $Z$  be the random variable satisfying  $\mathbb{P}(Z = 0) = \beta$  and  $\mathbb{P}(Z = 1) = 1 - \beta$ , i.e., the event  $Z = 0$  and  $Z = 1$  corresponds to that the state  $s$  is sampled from  $\nu_{\pi}$  and  $\zeta$ , respectively. For any random variable  $Y$ , following the law of total expectation, we know that
$$
\begin{array}{l} \mathbb {E} _ {\mu_ {\pi}} [ Y ] = \mathbb {E} [ \mathbb {E} [ Y | Z ] ] = \mathbb {E} [ Y | Z = 0 ] \mathbb {P} (Z = 0) + \mathbb {E} [ Y | Z = 1 ] \mathbb {P} (Z = 1) \\ = \beta \mathbb {E} [ Y | Z = 0 ] + (1 - \beta) \mathbb {E} [ Y | Z = 1 ] \\ = \beta \mathbb {E} _ {\nu_ {\pi}} [ Y ] + (1 - \beta) \mathbb {E} _ {\zeta} [ Y ]. \tag {B.29} \\ \end{array}
$$
Therefore, we have from (B.28) that
$$
\begin{array}{l} b _ {t} \leq \mathbb {E} _ {\widehat {s} _ {0, n} \sim \mu_ {\pi}} \left[ \left\| \mathbb {E} _ {s _ {0} \sim \zeta} \left[ \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (s _ {i}, a _ {i}) + \gamma^ {h} \cdot \nabla_ {\theta} V ^ {\pi_ {\theta}} (s _ {h}) - \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (\widehat {s} _ {i, n}, \widehat {a} _ {i, n}) \right. \right. \right. \\ \left. - \gamma^ {h} \cdot \nabla_ {\theta} \widehat {V} _ {t} (\widehat {s} _ {h, n}) \right] \Bigg \| _ {2} \Bigg ] \\ \end{array}
$$
$$
\begin{array}{l} \leq \beta \mathbb {E} _ {\widehat {s} _ {0, n} \sim \nu_ {\pi}} \left[ \left\| \mathbb {E} _ {s _ {0} \sim \zeta} \left[ \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (s _ {i}, a _ {i}) + \gamma^ {h} \cdot \nabla_ {\theta} V ^ {\pi_ {\theta}} (s _ {h}) - \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (\widehat {s} _ {i, n}, \widehat {a} _ {i, n}) \right. \right. \right. \\ \left. - \gamma^ {h} \cdot \nabla_ {\theta} \widehat {V} _ {t} (\widehat {s} _ {h, n}) \right] \Bigg \| _ {2} \Bigg ] + (1 - \beta) \mathbb {E} _ {\widehat {s} _ {0, n} \sim \zeta} \Bigg [ \Bigg \| \mathbb {E} _ {s _ {0} \sim \zeta} \Bigg [ \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (s _ {i}, a _ {i}) + \gamma^ {h} \cdot \nabla_ {\theta} V ^ {\pi_ {\theta}} (s _ {h}) \\ \left. - \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r \left(\widehat {s} _ {i, n}, \widehat {a} _ {i, n}\right) - \gamma^ {h} \cdot \nabla_ {\theta} \widehat {V} _ {t} \left(\widehat {s} _ {h, n}\right) \right] \Bigg \| _ {2} \Bigg ], \tag {B.30} \\ \end{array}
$$
where the first inequality holds since  $\| \mathbb{E}[\cdot ]\| _2\leq \mathbb{E}[\| \cdot \| _2]$  and the second inequality holds due to (B.29).
Using the result from Lemma B.3, we know that
$$
\begin{array}{l} \mathbb {E} _ {s _ {0} \sim \zeta} \left[ \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (s _ {i}, a _ {i}) + \gamma^ {h} \cdot \nabla_ {\theta} V ^ {\pi_ {\theta}} (s _ {h}) - \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (\widehat {s} _ {i, n}, \widehat {a} _ {i, n}) - \gamma^ {h} \cdot \nabla_ {\theta} \widehat {V} _ {t} (\widehat {s} _ {h, n}) \right] \\ = \underbrace {\mathbb {E} _ {s _ {0} \sim \zeta} \left[ \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (s _ {i} , a _ {i}) - \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (\widehat {s} _ {i , n} , \widehat {a} _ {i , n}) \right]} _ {B _ {r}} \\ + \underbrace {\gamma^ {h} \mathbb {E} _ {(s _ {h} , a _ {h}) \sim \overline {{\sigma}} _ {1}} \left[ \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a _ {h}} \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} + \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s _ {h}} \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} - \nabla_ {\theta} \widehat {V} _ {t} (\widehat {s} _ {h , n}) \right]} _ {B _ {v}}. \\ \end{array}
$$
Here, the shorthand notation  $B_r$  denotes the bias introduced by the  $h$ -step model expansion and  $B_v$  denotes the bias introduced by using a critic for tail estimation. Then we may rewrite (B.30) as
$$
\begin{array}{l} b _ {t} \leq \beta \cdot \mathbb {E} _ {\widehat {s} _ {0, n} \sim \nu_ {\pi}} \left[ \left\| B _ {r} + B _ {v} \right\| _ {2} \right] + (1 - \beta) \cdot \mathbb {E} _ {\widehat {s} _ {0, n} \sim \zeta} \left[ \left\| B _ {r} + B _ {v} \right\| _ {2} \right] \\ \leq \left(\beta \cdot \mathbb {E} _ {\widehat {s} _ {0, n} \sim \nu_ {\pi}} \left[ \left\| B _ {r} \right\| _ {2} \right] + (1 - \beta) \cdot \mathbb {E} _ {\widehat {s} _ {0, n} \sim \zeta} \left[ \left\| B _ {r} \right\| _ {2} \right]\right) \\ + \left(\beta \cdot \mathbb {E} _ {\widehat {s} _ {0, n} \sim \nu_ {\pi}} \left[ \| B _ {v} \| _ {2} \right] + (1 - \beta) \cdot \mathbb {E} _ {\widehat {s} _ {0, n} \sim \zeta} \left[ \| B _ {v} \| _ {2} \right]\right). \tag {B.31} \\ \end{array}
$$
For the first term on the right-hand side of (B.31), we have
$$
\begin{array}{l} \beta \cdot \mathbb {E} _ {\hat {s} _ {0, n} \sim \nu_ {\pi}} \left[ \left\| B _ {r} \right\| _ {2} \right] + (1 - \beta) \cdot \mathbb {E} _ {\hat {s} _ {0, n} \sim \zeta} \left[ \left\| B _ {r} \right\| _ {2} \right] \\ = \beta \cdot \mathbb {E} _ {\widehat {s} _ {0, n} \sim \nu_ {\pi}} \left[ \left\| \mathbb {E} _ {s _ {0} \sim \zeta} \left[ \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r \left(s _ {i}, a _ {i}\right) - \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r \left(\widehat {s} _ {i, n}, \widehat {a} _ {i, n}\right) \right] \right\| _ {2} \right] + (1 - \beta). \\ \mathbb {E} _ {\widehat {s} _ {0, n} \sim \nu_ {\pi}} \left[ \left\| \mathbb {E} _ {s _ {0} \sim \zeta} \left[ \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (s _ {i}, a _ {i}) - \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r (\widehat {s} _ {i, n}, \widehat {a} _ {i, n}) \right] \right\| _ {2} \right] \cdot \left\{\mathbb {E} _ {\nu_ {\pi}} \left[ \left(\frac {\mathrm {d} \zeta}{\mathrm {d} \nu_ {\pi}} (s)\right) ^ {2} \right] \right\} ^ {1 / 2} \\ \leq \left(\beta + \kappa \cdot (1 - \beta)\right) \cdot \mathbb {E} _ {\widehat {s} _ {0, n} \sim \nu_ {\pi}} \left[ \left\| \mathbb {E} _ {s _ {0} \sim \zeta} \left[ \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r \left(s _ {i}, a _ {i}\right) - \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r \left(\widehat {s} _ {i, n}, \widehat {a} _ {i, n}\right) \right] \right\| _ {2} \right] \\ \leq \kappa (\beta + \kappa \cdot (1 - \beta)) \cdot \mathbb {E} _ {s _ {0} \sim \nu_ {\pi}, \widehat {s} _ {0, n} \sim \nu_ {\pi}} \left[ \left\| \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r \left(s _ {i}, a _ {i}\right) - \nabla_ {\theta} \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \cdot r \left(\widehat {s} _ {i, n}, \widehat {a} _ {i, n}\right) \right\| _ {2} \right], \tag {B.32} \\ \end{array}
$$
where the first and second inequalities follow from the definition of  $\kappa$  in Proposition 5.6.
Similarly, for the second term on the right-hand side of (B.31), we have
$$
\begin{array}{l} \beta \cdot \mathbb {E} _ {\hat {s} _ {0, n} \sim \nu_ {\pi}} \left[ \left\| B _ {v} \right\| _ {2} \right] + (1 - \beta) \cdot \mathbb {E} _ {\hat {s} _ {0, n} \sim \zeta} \left[ \left\| B _ {v} \right\| _ {2} \right] \\ = \beta \cdot \mathbb {E} _ {\widehat {s} _ {0, n} \sim \nu_ {\pi}} \left[ \gamma^ {h} \cdot \left\| \mathbb {E} _ {(s _ {h}, a _ {h}) \sim \overline {{\sigma}} _ {1}} \left[ \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a _ {h}} \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} + \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s _ {h}} \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} - \nabla_ {\theta} \widehat {V} _ {t} (\widehat {s} _ {h, n}) \right] \right\| _ {2} \right] \\ \left. \right. + (1 - \beta) \cdot \mathbb {E} _ {\widehat {s} _ {0, n} \sim \zeta} \left[ \gamma^ {h} \cdot \left\| \mathbb {E} _ {(s _ {h}, a _ {h}) \sim \overline {{\sigma}} _ {1}} \left[ \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a _ {h}} \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} + \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s _ {h}} \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} - \nabla_ {\theta} \widehat {V} _ {t} (\widehat {s} _ {h, n}) \right]\right\| _ {2} \right] \\ \leq \left(\beta + \kappa \cdot (1 - \beta)\right) \gamma^ {h} \cdot \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \overline {{\sigma}} _ {1}, \widehat {s} _ {0, n} \sim \nu_ {\pi}} \left[ \left\| \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a _ {h}} \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} + \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s _ {h}} \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} - \nabla_ {\theta} \widehat {V} _ {t} (\widehat {s} _ {h, n}) \right\| _ {2} \right] \\ = (\beta + \kappa \cdot (1 - \beta)) \gamma^ {h}. \\ \end{array}
$$
$$
\mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \bar {\sigma} _ {1}, \left(\widehat {s} _ {h, n}, \widehat {a} _ {h, n}\right) \sim \widehat {\sigma} _ {1}} \left[ \left\| \frac {\partial Q ^ {\pi_ {\theta}}}{\partial a _ {h}} \cdot \frac {\mathrm {d} a _ {h}}{\mathrm {d} \theta} + \frac {\partial Q ^ {\pi_ {\theta}}}{\partial s _ {h}} \cdot \frac {\mathrm {d} s _ {h}}{\mathrm {d} \theta} - \nabla_ {\theta} \widehat {Q} _ {t} \left(\widehat {s} _ {h, n}, \widehat {a} _ {h, n}\right) \right\| _ {2} \right]. \tag {B.33}
$$
Plugging (B.32) and (B.33) into (B.31) completes the proof.
Lemma B.5 (Lipschitz Value Function [47] Theorem 1). Under Assumption 5.3, for  $\gamma L_{f}(1 + L_{\pi}) < 1$ , then the state-action value function is  $L_{Q}$ -Lipschitz continuous, such that for any policy  $\pi_{\theta}$ , state  $s_1, s_2 \in S$  and action  $a_1, a_2 \in \mathcal{A}$ ,  $|Q^{\pi_{\theta}}(s_1, a_1) - Q^{\pi_{\theta}}(s_2, a_2)| \leq L_Q \cdot \| (s_1 - s_2, a_1 - a_2) \|_2$ , and
$$
L _ {Q} = L _ {r} / \left(1 - \gamma L _ {f} \left(1 + L _ {\pi}\right)\right).
$$
B.4 Proof of Proposition 5.7
Proof. When  $\gamma \approx 1$ , we have
$$
\frac {1 - \gamma^ {h}}{1 - \gamma} = \sum_ {i = 0} ^ {h - 1} \gamma^ {i} \approx h, \quad \frac {\gamma^ {h}}{1 - \gamma} = \frac {1}{1 - \gamma} - \frac {1 - \gamma^ {h}}{1 - \gamma} \approx \frac {1}{1 - \gamma} - h.
$$
We denote by  $H = 1 / (1 - \gamma) = \sum_{i=0}^{\infty} \gamma^{i}$  the effective task horizon.
To find the optimal unroll length  $h^*$  that minimizes the upper bound of the convergence, we define  $g(h)$  as follows,
$$
g (h) = c \cdot (2 \delta \cdot b _ {t} ^ {\prime} + \frac {\eta}{2} \cdot v _ {t} ^ {\prime 2}) + b _ {t} ^ {\prime 2} + v _ {t} ^ {\prime 2}.
$$
Here,  $v_{t}^{\prime 2}$  and  $b_{t}^{\prime}$  are the leading terms in the variance, bias bound (i.e., (B.12) and (B.24)) when  $L_{f}, L_{\widehat{f}}$ , and  $L_{\pi}$  are less than or equal to 1. Formally,  $v_{t}^{\prime} = h^{3}$  and  $b_{t}^{\prime} = h^{3}\epsilon_{f,t} + h(H - h)^{2}\epsilon_{v,t}$ . We consider the terms that are only dependent on  $h, H, \epsilon_{f,t}$ , and  $\epsilon_{v,t}$  to simplify the analysis and determine the order of  $h^{*}$ .
Our first problem is to find the optimal model unroll  $h^*$  that minimizes  $g(h)$ . We notice that  $g(h)$  increases monotonically with respect to  $b_t'$  and  $v_t'$  when they are non-negative. This further simplifies the problem to find
$$
h ^ {\prime *} = \underset {h} {\operatorname {a r g m i n}} b _ {t} ^ {\prime} + c ^ {\prime} v _ {t} ^ {\prime} = \underset {h} {\operatorname {a r g m i n}} \underbrace {h ^ {3} \left(\epsilon_ {f , t} + c ^ {\prime}\right) + h (H - h) ^ {2} \epsilon_ {v , t}} _ {g _ {1} (h)}, \tag {B.34}
$$
where  $c'$  is some constant that does not affect the order of  $h'/\ast$ .
By taking the derivative of the right-hand side of (B.34) with respect to  $h$  and setting it to zero, we obtain
$$
\frac {\partial}{\partial h} g _ {1} (h) = 3 h ^ {2} \cdot \left(\epsilon_ {f, t} + c ^ {\prime}\right) + \left(3 h ^ {2} - 4 H h + H ^ {2}\right) \cdot \epsilon_ {v, t} = 0. \tag {B.35}
$$
Solve the above quadratic equation with respect to  $h$ , we have the two non-negative roots  $h_1^{\prime *}$  and  $h_2^{\prime *}$  as follows,
$$
h _ {1} ^ {\prime *} = \frac {4 H \epsilon_ {v , t} + \sqrt {(4 H \epsilon_ {v , t}) ^ {2} - 1 2 c _ {1} \epsilon_ {v , t} H ^ {2}}}{6 c _ {1}}, h _ {2} ^ {\prime *} = \frac {4 H \epsilon_ {v , t} - \sqrt {(4 H \epsilon_ {v , t}) ^ {2} - 1 2 c _ {1} \epsilon_ {v , t} H ^ {2}}}{6 c _ {1}},
$$
where we define  $c_{1} = \epsilon_{f,t} + \epsilon_{v,t} + c^{\prime}$
Now we study the resulting two cases. If  $(4H\epsilon_{v,t})^2 - 12c_1\epsilon_{v,t}H^2 \geq 0$ , we have
$$
h ^ {*} = h _ {1} ^ {*} = O \left(\epsilon_ {v, t} / \left(\epsilon_ {f, t} + \epsilon_ {v, t}\right) \cdot H\right).
$$
We can verify that  $h_1^{\prime *}$  is indeed the minimum by calculating the second-order derivative at  $h_1^{\prime *}$  as follows,
$$
\begin{array}{l} \frac {\partial^ {2} g _ {1} \left(h _ {1} ^ {\prime *}\right)}{\partial h ^ {2}} = \frac {4 H \epsilon_ {v , t} + \sqrt {\left(4 H \epsilon_ {v , t}\right) ^ {2} - 4 c _ {1} \cdot H ^ {2}}}{6 c _ {1}} * 6 \left(\epsilon_ {f, t} + \epsilon_ {v, t} + c ^ {\prime}\right) - 4 H \epsilon_ {v, t} \\ = \sqrt {(4 H \epsilon_ {v , t}) ^ {2} - 4 c _ {1} \cdot H ^ {2}} > 0. \\ \end{array}
$$
The other case is  $(4H\epsilon_{v,t})^2 - 12c_1\epsilon_{v,t}H^2 < 0$ . When this happens, (B.35) does not have a real solution  $h^{t*}$  and we set  $h^{*}$  to 0. This concludes the proof of Proposition 5.7.
B.5 Proof of Corollary 5.9
Proof. We let the learning rate  $\eta = 1 / \sqrt{T}$ . Then for  $T \geq 4L^2$ , we have  $c = (\eta - L\eta^2)^{-1} \leq 2\sqrt{T}$  and  $L\eta \leq 1 / 2$ . By setting  $N = O(\sqrt{T})$ , we obtain
$$
\begin{array}{l} \min  _ {t \in [ T ]} \mathbb {E} \left[ \left\| \nabla_ {\theta} J (\pi_ {\theta_ {t}}) \right\| _ {2} ^ {2} \right] \leq \frac {4}{T} \cdot \left(\sum_ {t = 0} ^ {T - 1} c \cdot (2 \delta \cdot b _ {t} + \frac {\eta}{2} \cdot v _ {t}) + b _ {t} ^ {2} + v _ {t}\right) + \frac {4 c}{T} \cdot \mathbb {E} \left[ J (\pi_ {\theta_ {T}}) - J (\pi_ {\theta_ {1}}) \right] \\ \leq \frac {4}{T} \left(\sum_ {t = 0} ^ {T - 1} 4 \sqrt {T} \delta \cdot b _ {t} + b _ {t} ^ {2} + 2 v _ {t}\right) + \frac {8}{\sqrt {T}} \cdot \mathbb {E} \left[ J \left(\pi_ {\theta_ {T}}\right) - J \left(\pi_ {\theta_ {1}}\right) \right] \\ \leq \frac {4}{T} \left(\sum_ {t = 0} ^ {T - 1} 4 \sqrt {T} \delta \cdot b _ {t} + b _ {t} ^ {2}\right) + O \left(1 / \sqrt {T}\right) \\ \leq \frac {1 6 \delta}{\sqrt {T}} \varepsilon (T) + \frac {4}{T} \varepsilon^ {2} (T) + O \big (1 / \sqrt {T} \big). \\ \end{array}
$$
C.1 Implementations and Comparisons with More RL Baselines
For the model-based baseline Model-Based Policy Optimization (MBPO) [28], we use the implementation in the Mbrl-lib [45]. For all other model-free baselines, we use the implementations in Tianshou [62] that have state-of-the-art results.
We observe that the RP-DP has competitive performance in all the evaluation tasks compared to the popular baselines, suggesting the importance of studying model-based RP PGMs. In experiments, we implement RP-DR as the on-policy SVG(1) [25]. We observe that the training can be unstable when using the off-policy SVG implementation, which requires a carefully chosen policy update rate as well as a proper size of the experience replay buffer. This is because when the learning rate is large, the magnitude of the inferred policy noise (from the previous data samples in the experience replay) can be huge. Implementing an on-policy version of RP-DR can avoid such an issue, following [25]. This, however, can degrade the performance of RP-DR compared to the off-policy RP-DP algorithm in several tasks. We conjecture that implementing the off-policy version of RP-DR can boost its performance, which requires techniques to stabilize training and we leave it as future work. For RP-DP, we implement it as Model-Augmented Actor-Critic (MAAC) [12] with entropy regularization [23], as suggested by [4].  $\mathrm{RP}(0)$  represents setting  $h = 0$  in the RP PGM formulas [4], which is a model-free algorithm that is a stochastic counterpart of deterministic policy gradients.
For model-free baselines, we compare with Likelihood Ratio (LR) policy gradient methods (c.f. (2.2)), including REINFORCE [56], Natural Policy Gradient (NPG) [31], Advantage Actor Critic (A2C), Actor Critic using Kronecker-Factored Trust Region (ACKTR) [65], and Proximal Policy Optimization (PPO) [49]. We also evaluate algorithms that are built upon DDPG [34], including Soft Actor-Critic (SAC) [23] and Twin Delayed Deep Deterministic policy gradient (TD3) [19].
C.2 Implementation of Spectral Normalization
In experiments, we use Multilayer Perceptrons (MLPs) for the critic, policy, and model. Besides, we adopt Gaussian dynamical models and policies as the source of stochasticity. To test the benefit of smooth function approximations in model-based RP policy gradient algorithms, spectral normalization is applied to all layers of the policy MLP and all except the final layers of the model MLP. The number of layers for the policy and the dynamics model is 4 and 5, respectively.
Our code is based on PyTorch [44], which has an out-of-the-shelf implementation of spectral normalization. Thus, applying SN to the MLP is pretty simple and no additional lines of code are needed. Specifically, we only need to import and apply SN to each layer:
from torch.nn.utils parametrizations import spectral_norm layer = [spectral_norm(nn.Linear(in_dim, hidden_dim)), nn.ReLU())]
C.3 Ablation on Spectral Normalization
In this section, we conduct ablation studies on the spectral normalization applied to model-based RP PGMs. Specifically, we aim to answer the following two questions: (1) What are the effects of SN when applied to different NN components of model-based RP PGMs? (2) Does SN improve other MBRL algorithms by smoothing the models?
What are the effects of SN when applied to different NN components of model-based RP PGMs? We study the following NN components that spectral normalization is applied to: both the model and the policy (default setting as suggested by our theory); only the model; only the policy; no SN is applied (vanilla setting). The results are shown in Figure 8.
Figure 8: Ablation on the effects of spectral normalization when applied to different NN components of model-based RP PGMs.
We observe in Fig. 8 that for both the hopper and the walker2d tasks, applying SN to the model and policy simultaneously achieves the best performance, which supports our theoretical results. Besides, learning a smooth transition kernel by applying SN to the neural network model only is slightly better than only applying SN to the policy. At the same time, the vanilla implementation of model-based RP PGM fails to give acceptable results.
Does SN improve other MBRL algorithms by smoothing the models? We have established that smoothness regularization, such as SN, in model-based RP PGMs can reduce the gradient variance and improve their convergence and performance. However, it is not necessarily the case for other model-based RL methods. In this part, we investigate whether SN can improve previous MBRL algorithms due to a smoothed model. Specifically, we evaluate MBPO [28], a popular MBRL algorithm when SN is added to different numbers of layers in the model neural network. The results are shown in Figure 9. We observe that SN has a negative influence when applied, which is in contrast to our findings that SN is beneficial to RP PGMs.
Figure 9: Ablation on the effect of SN in other MBRL algorithms. Spectral normalization has a negative effect when applied to MBPO, indicating that SN by smoothing the model, does not necessarily lead to better gradient estimation or improve the performance for MBRL methods other than model-based RP PGMs.
C.4 Ablation on Different Model Learners