SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \bar {f} (w, \theta_ {1}) - \bar {f} (w, \theta_ {2}) \right\| \\ = \left\| \sum_ {s, a} \mu_ {\theta_ {1}} (s, a) \sum_ {s ^ {\prime}, a ^ {\prime}} P _ {\text {e n v}} \left(s ^ {\prime} \mid s, a\right) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {1}\right) f (O, w) \right. \\ \left. - \sum_ {s, a} \mu_ {\theta_ {2}} (s, a) \sum_ {s ^ {\prime}, a ^ {\prime}} P _ {\text {e n v}} \left(s ^ {\prime} \mid s, a\right) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {2}\right) f (O, w) \right\| \\ = \left\| \sum_ {s, a} \sum_ {s ^ {\prime}, a ^ {\prime}} \left[ \mu_ {\theta_ {1}} (s, a) P _ {\text {e n v}} \left(s ^ {\prime} \mid s, a\right) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {1}\right) - \mu_ {\theta_ {2}} (s, a) P _ {\text {e n v}} \left(s ^ {\prime} \mid s, a\right) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {2}\right) \right] f (O, w) \right\| \\ \leq \sum_ {s, a, s ^ {\prime}, a ^ {\prime}} P _ {\text {e n v}} \left(s ^ {\prime} \mid s, a\right) \left| \mu_ {\theta_ {1}} (s, a) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {1}\right) - \mu_ {\theta_ {2}} (s, a) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {2}\right) \right| \cdot | | f (O, w) | | \\ \leq U _ {f} \sum_ {s, a, s ^ {\prime}, a ^ {\prime}} P _ {\mathrm {e n v}} (s ^ {\prime} | s, a) \left| \mu_ {\theta_ {1}} (s, a) \pi (a ^ {\prime} | s ^ {\prime}, \theta_ {1}) - \mu_ {\theta_ {2}} (s, a) \pi (a ^ {\prime} | s ^ {\prime}, \theta_ {2}) \right|, \\ \end{array}
$$
where the last line is by Lemma C.15. Further notice that
$$
\begin{array}{l} \left| \mu_ {\theta_ {1}} (s, a) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {1}\right) - \mu_ {\theta_ {2}} (s, a) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {2}\right) \right| \\ \leq \left| \mu_ {\theta_ {1}} (s, a) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {1}\right) - \mu_ {\theta_ {1}} (s, a) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {2}\right) \right| \\ + \left| \mu_ {\theta_ {1}} (s, a) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {2}\right) - \mu_ {\theta_ {2}} (s, a) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {2}\right) \right| \\ = \mu_ {\theta_ {1}} (s, a) | \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {1}\right) - \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {2}\right) | + \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {2}\right) \left| \mu_ {\theta_ {1}} (s, a) - \mu_ {\theta_ {2}} (s, a) \right|. \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {s, a, s ^ {\prime}, a ^ {\prime}} P _ {\mathrm {e n v}} \left(s ^ {\prime} \mid s, a\right) \left| \mu_ {\theta_ {1}} (s, a) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {1}\right) - \mu_ {\theta_ {2}} (s, a) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {2}\right) \right| \\ \leq \sum_ {s, a, s ^ {\prime}, a ^ {\prime}} P _ {\text {e n v}} \left(s ^ {\prime} \mid s, a\right) \mu_ {\theta_ {1}} (s, a) \left| \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {1}\right) - \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {2}\right) \right| \\ + \sum_ {s, a, s ^ {\prime}, a ^ {\prime}} P _ {\text {e n v}} \left(s ^ {\prime} \mid s, a\right) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {2}\right) \left| \mu_ {\theta_ {1}} (s, a) - \mu_ {\theta_ {2}} (s, a) \right| \\ \leq L _ {\pi} | | \theta_ {1} - \theta_ {2} | | \sum_ {s, a, s ^ {\prime}, a ^ {\prime}} P _ {\mathrm {e n v}} (s ^ {\prime} | s, a) \mu_ {\theta_ {1}} (s, a) \\ + \sum_ {s, a, s ^ {\prime}, a ^ {\prime}} P _ {\text {e n v}} \left(s ^ {\prime} \mid s, a\right) \pi \left(a ^ {\prime} \mid s ^ {\prime}, \theta_ {2}\right) \left| \mu_ {\theta_ {1}} (s, a) - \mu_ {\theta_ {2}} (s, a) \right| \\ = n _ {a} L _ {\pi} \left\| \theta_ {1} - \theta_ {2} \right\| + \left\| \mu_ {\theta_ {1}} - \mu_ {\theta_ {2}} \right\| _ {1} \\ \leq \left(2 + \tau_ {\text {m i x}} + C \frac {1}{1 - \beta}\right) n _ {a} L _ {\pi} | | \theta_ {1} - \theta_ {2} | |, \\ \end{array}
$$
where the last line is by Lemma C.9. This implies
$$
\begin{array}{l} \left\| \bar {f} (w, \theta_ {1}) - \bar {f} (w, \theta_ {2}) \right\| \leq U _ {f} \left(2 + \tau_ {\text {m i x}} + C \frac {1}{1 - \beta}\right) n _ {a} L _ {\pi} \| \theta_ {1} - \theta_ {2} \| \\ = L _ {Q} U _ {\delta} \left(2 + \tau_ {\mathrm {m i x}} + C \frac {1}{1 - \beta}\right) n _ {a} L _ {\pi} | | \theta_ {1} - \theta_ {2} | |. \\ \end{array}
$$
Lemma C.17. Denote  $F(O, w, \theta) = (w - \hat{w}_{\theta}^{*})^{T} [f(O, w) - \bar{f}(w, \theta)]$ . The following results hold:
a.  $F(O,w,\theta)$  is  $L_{F_{\theta}}$  -Lipschitz with respect to  $\theta$
$$
\left| F (O, w, \theta_ {1}) - F (O, w, \theta_ {2}) \right| \leq L _ {F _ {\theta}} | | \theta_ {1} - \theta_ {2} | |,
$$
$$
L _ {F _ {\theta}} = 2 \left(U _ {f} L _ {w} + \sigma_ {w} L _ {\bar {f}}\right).
$$
b.  $F(O,w,\theta)$  is  $L_{F_w}$ -Lipschitz with respect to  $w$ ,
$$
| F (O, w _ {1}, \theta) - F (O, w _ {2}, \theta) | \leq L _ {F _ {w}} | | w _ {1} - w _ {2} | |,
$$
$$
L _ {F _ {w}} = 4 \sigma_ {w} L _ {f} + 2 U _ {f}.
$$
c. Conditioned on  $\theta_{t - \tau_{\mathrm{mix}}}$  and  $s_{t - \tau_{\mathrm{mix}} + 1}$
$$
\begin{array}{l} \left| \mathbb {E} \left[ F (O _ {t}, w _ {t - \tau_ {\mathrm {m i x}}}, \theta_ {t - \tau_ {\mathrm {m i x}}}) - F (\tilde {O} _ {t}, w _ {t - \tau_ {\mathrm {m i x}}}, \theta_ {t - \tau_ {\mathrm {m i x}}}) | \theta_ {t - \tau_ {\mathrm {m i x}}}, s _ {t - \tau_ {\mathrm {m i x}} + 1} \right] \right| \\ \leq 4 \sigma_ {w} U _ {f} n _ {a} L _ {\pi} \sum_ {i = t - \tau_ {\mathrm {m i x}}} ^ {t} \mathbb {E} \left[ \| \theta_ {i} - \theta_ {t - \tau_ {\mathrm {m i x}}} \| | \theta_ {t - \tau_ {\mathrm {m i x}}}, s _ {t - \tau_ {\mathrm {m i x}} + 1} \right]. \\ \end{array}
$$
d. Conditioned on  $\theta_{t - \tau_{\mathrm{mix}}}$  and  $s_{t - \tau_{\mathrm{mix}} + 1}$
$$
\left| \mathbb {E} \left[ F (\tilde {O} _ {t}, w _ {t - \tau_ {\mathrm {m i x}}}, \theta_ {t - \tau_ {\mathrm {m i x}}}) | \theta_ {t - \tau_ {\mathrm {m i x}}}, s _ {t - \tau_ {\mathrm {m i x}} + 1} \right] \right| \leq 2 \sigma_ {w} U _ {f} C \beta^ {\tau_ {\mathrm {m i x}} - 1}.
$$
Proof. First, we observe that
$$
\begin{array}{l} \left| F (O, w, \theta_ {1}) - F (O, w, \theta_ {2}) \right| \\ = \left| \left(w - \hat {w} _ {\theta_ {1}} ^ {*}\right) ^ {T} [ f (O, w) - \bar {f} (w, \theta_ {1}) ] - \left(w - \hat {w} _ {\theta_ {2}} ^ {*}\right) ^ {T} [ f (O, w) - \bar {f} (w, \theta_ {2}) ] \right| \\ \leq \left| \left(w - \hat {w} _ {\theta_ {1}} ^ {*}\right) ^ {T} \left[ f (O, w) - \bar {f} (w, \theta_ {1}) \right] - \left(w - \hat {w} _ {\theta_ {2}} ^ {*}\right) ^ {T} \left[ f (O, w) - \bar {f} (w, \theta_ {1}) \right] \right| \\ + \left| \left(w - \hat {w} _ {\theta_ {2}} ^ {*}\right) ^ {T} \left[ f (O, w) - \bar {f} (w, \theta_ {1}) \right] - \left(w - \hat {w} _ {\theta_ {2}} ^ {*}\right) ^ {T} \left[ f (O, w) - \bar {f} (w, \theta_ {2}) \right] \right| \\ = \underbrace {\left| \left(\hat {w} _ {\theta_ {2}} ^ {*} - \hat {w} _ {\theta_ {1}} ^ {*}\right) ^ {T} \left[ f (O , w) - \bar {f} (w , \theta_ {1}) \right] \right|} _ {I _ {1}} + \underbrace {\left| \left(w - \hat {w} _ {\theta_ {2}} ^ {*}\right) ^ {T} \left[ \bar {f} (w , \theta_ {2}) - \bar {f} (w , \theta_ {1}) \right] \right|} _ {I _ {2}}. \\ \end{array}
$$
For  $I_{1}$ , we have
$$
\left| \left(\hat {w} _ {\theta_ {2}} ^ {*} - \hat {w} _ {\theta_ {1}} ^ {*}\right) ^ {T} \left[ f (O, w) - \bar {f} (w, \theta_ {1}) \right] \right| \leq 2 U _ {f} L _ {w} | | \theta_ {1} - \theta_ {2} | |,
$$
which is by Lemma C.15 and Assumption 2.6.
For  $I_{2}$ , we have
$$
\left| \left(w - \hat {w} _ {\theta_ {2}} ^ {*}\right) ^ {T} \left[ \bar {f} \left(w, \theta_ {2}\right) - \bar {f} \left(w, \theta_ {1}\right) \right] \right| \leq 2 \sigma_ {w} L _ {\bar {f}} | | \theta_ {1} - \theta_ {2} | |,
$$
which is by Lemma C.16. Combining the above two facts we end the proof of the first part.
For the second part,
$$
\begin{array}{l} \left| F (O, w _ {1}, \theta) - F (O, w _ {2}, \theta) \right| \\ = \left| \left(w _ {1} - \hat {w} _ {\theta} ^ {*}\right) ^ {T} [ f (O, w _ {1}) - \bar {f} (w _ {1}, \theta) ] - \left(w _ {2} - \hat {w} _ {\theta} ^ {*}\right) ^ {T} [ f (O, w _ {2}) - \bar {f} (w _ {2}, \theta) ] \right| \\ \leq \left| \left(w _ {1} - \hat {w} _ {\theta} ^ {*}\right) ^ {T} [ f (O, w _ {1}) - \bar {f} (w _ {1}, \theta) ] - \left(w _ {1} - \hat {w} _ {\theta} ^ {*}\right) ^ {T} [ f (O, w _ {2}) - \bar {f} (w _ {2}, \theta) ] \right| \\ + \left| \left(w _ {1} - \hat {w} _ {\theta} ^ {*}\right) ^ {T} [ f (O, w _ {2}) - \bar {f} (w _ {2}, \theta) ] - \left(w _ {2} - \hat {w} _ {\theta} ^ {*}\right) ^ {T} [ f (O, w _ {2}) - \bar {f} (w _ {2}, \theta) ] \right| \\ = \left| \left(w _ {1} - \hat {w} _ {\theta} ^ {*}\right) ^ {T} \left[ \left(f (O, w _ {1}) - f (O, w _ {2})\right) - \left(\bar {f} (w _ {1}, \theta) - \bar {f} (w _ {2}, \theta)\right) \right] \right| \\ + \left| (w _ {1} - w _ {2}) ^ {T} [ f (O, w _ {2}) - \bar {f} (w _ {2}, \theta) ] \right| \\ \leq 4 \sigma_ {w} L _ {f} | | w _ {1} - w _ {2} | | + 2 U _ {f} | | w _ {1} - w _ {2} | |, \\ \end{array}
$$
where the last line is by Lemma C.15 and Lemma C.16.
For the third part, conditioned on  $\theta_{t - \tau_{\mathrm{mix}}}$  and  $s_{t - \tau_{\mathrm{mix}} + 1}$ ,
$$
\begin{array}{l} \left| \mathbb {E} \left[ F (O _ {t}, w _ {t - \tau_ {\mathrm {m i x}}}, \theta_ {t - \tau_ {\mathrm {m i x}}}) - F (\tilde {O} _ {t}, w _ {t - \tau_ {\mathrm {m i x}}}, \theta_ {t - \tau_ {\mathrm {m i x}}}) \right] \right| \\ = \left| \mathbb {E} \left[ \left(w _ {t - \tau_ {\mathrm {m i x}}} - \hat {w} _ {\theta_ {t - \tau_ {\mathrm {m i x}}}} ^ {*}\right) ^ {T} \left[ f (O _ {t}, w _ {t - \tau_ {\mathrm {m i x}}}) - \bar {f} (w _ {t - \tau_ {\mathrm {m i x}}}, \theta_ {t - \tau_ {\mathrm {m i x}}}) \right] \right. \right. \\ \left. - \left(w _ {t - \tau_ {\mathrm {m i x}}} - \hat {w} _ {\theta_ {t - \tau_ {\mathrm {m i x}}}} ^ {*}\right) ^ {T} \left[ f (\tilde {O} _ {t}, w _ {t - \tau_ {\mathrm {m i x}}}) - \bar {f} (w _ {t - \tau_ {\mathrm {m i x}}}, \theta_ {t - \tau_ {\mathrm {m i x}}}) \right] \right] \\ = \left| \mathbb {E} \left[ \left(w _ {t - \tau_ {\operatorname {m i x}}} - \hat {w} _ {\theta_ {t - \tau_ {\operatorname {m i x}}}} ^ {*}\right) ^ {T} \left[ f (O _ {t}, w _ {t - \tau_ {\operatorname {m i x}}}) - f (\tilde {O} _ {t}, w _ {t - \tau_ {\operatorname {m i x}}}) \right] \right] \right| \\ \leq 2 \sigma_ {w} \| \mathbb {E} \left[ f \left(O _ {t}, w _ {t - \tau_ {\operatorname {m i x}}}\right) - f \left(\tilde {O} _ {t}, w _ {t - \tau_ {\operatorname {m i x}}}\right) \right] \| \\ \leq 2 \sigma_ {w} U _ {f} | | P (O _ {t} \in \cdot) - P (\tilde {O} _ {t} \in \cdot) | | _ {1} \\ \leq 4 \sigma_ {w} U _ {f} n _ {a} L _ {\pi} \sum_ {i = t - \tau_ {\mathrm {m i x}}} ^ {t} \mathbb {E} \left[ \left\| \theta_ {i} - \theta_ {t - \tau_ {\mathrm {m i x}}} \right\| \right], \\ \end{array}
$$
where we use Lemma C.15 and Lemma C.10.
For the fourth part, we first denote  $O^{+} = (s^{+}, a^{+}, s^{+^{\prime}}, a^{+^{\prime}})$  such that  $(s^{+}, a^{+}) \sim \mu_{\theta_{t - \tau_{\mathrm{mix}}}}, s^{+^{\prime}} \sim P_{\mathrm{env}}(s^{\prime}|s,a)$  and  $a^{+^{\prime}} \sim \pi(a^{\prime}|s^{\prime}, \theta_{t - \tau_{\mathrm{mix}}})$ . Under this definition, we have
$$
\mathbb {E} \left[ F (O ^ {+}, w _ {t - \tau_ {\mathrm {m i x}}}, \theta_ {t - \tau_ {\mathrm {m i x}}}) | \theta_ {t - \tau_ {\mathrm {m i x}}}, s _ {t - \tau_ {\mathrm {m i x}} + 1} \right] = 0.
$$
By Assumption 2.8, we have
$$
| | P (\tilde {S} _ {t} \in \cdot | \theta_ {t - \tau_ {\mathrm {m i x}}}, s _ {t - \tau_ {\mathrm {m i x}} - 1}) - \mu_ {\theta_ {t - \tau_ {\mathrm {m i x}}} ^ {\prime}} ^ {\prime} | | _ {1} \leq C \beta^ {\tau_ {\mathrm {m i x}} - 1}.
$$
Hence, conditioned on  $\theta_{t - \tau_{\mathrm{mix}}}$  and  $s_{t - \tau_{\mathrm{mix}} + 1}$
$$
\begin{array}{l} \left| \mathbb {E} \left[ F (\tilde {O} _ {t}, w _ {t - \tau_ {\mathrm {m i x}}}, \theta_ {t - \tau_ {\mathrm {m i x}}}) \right] \right| \\ = \left| \mathbb {E} \left[ F \left(\tilde {O} _ {t}, w _ {t - \tau_ {\operatorname {m i x}}}, \theta_ {t - \tau_ {\operatorname {m i x}}}\right) - F \left(O ^ {+}, w _ {t - \tau_ {\operatorname {m i x}}}, \theta_ {t - \tau_ {\operatorname {m i x}}}\right) \right] \right| \\ = \left| \mathbb {E} \left[ (w _ {t - \tau_ {\mathrm {m i x}}} - \hat {w} _ {\theta_ {t - \tau_ {\mathrm {m i x}}}} ^ {*}) ^ {T} \left[ f (\tilde {O} _ {t}, w _ {t - \tau_ {\mathrm {m i x}}}) - f (O ^ {+}, w _ {t - \tau_ {\mathrm {m i x}}}) \right] \right] \right| \\ \leq 2 \sigma_ {w} U _ {f} | | P (\tilde {O} _ {t} \in \cdot) - P (O ^ {+} \in \cdot) | | _ {1}, \\ \end{array}
$$
where the last line is because of Lemma C.15. However,
$$
\begin{array}{l} \left\| P \left(\tilde {O} _ {t} \in \cdot\right) - P \left(O ^ {+} \in \cdot\right) \right\| _ {1} \\ = \sum_ {s, a, s ^ {\prime}, a ^ {\prime}} | P (\tilde {S} _ {t} = s, \tilde {A} _ {t} = a, \tilde {S} _ {t + 1} = s ^ {\prime}, \tilde {A} _ {t + 1} = a ^ {\prime}) - P (S _ {t} ^ {+} = s, A _ {t} ^ {+} = a, S _ {t + 1} ^ {+} = s ^ {\prime}, A _ {t + 1} ^ {+} = a ^ {\prime}) | \\ = \sum_ {s, a, s ^ {\prime}, a ^ {\prime}} | P (\tilde {S} _ {t} = s) \pi (a | s, \theta_ {t - \tau_ {\mathrm {m i x}}}) P _ {\mathrm {e n v}} (s ^ {\prime} | s, a) \pi (a ^ {\prime} | s ^ {\prime}, \theta_ {t - \tau_ {\mathrm {m i x}}}) \\ - P \left(S _ {t} ^ {+} = s\right) \pi \left(a | s, \theta_ {t - \tau_ {\mathrm {m i x}}}\right) P _ {\mathrm {e n v}} \left(s ^ {\prime} | s, a\right) \pi \left(a ^ {\prime} | s ^ {\prime}, \theta_ {t - \tau_ {\mathrm {m i x}}}\right) \\ = \left\| P \left(\tilde {S} _ {t} \in \cdot\right) - P \left(S _ {t} ^ {+} \in \cdot\right) \right\| _ {1} \\ \leq C \beta^ {\tau_ {\mathrm {m i x}} - 1}. \\ \end{array}
$$
Hence, conditioned on  $\theta_{t - \tau_{\mathrm{mix}}}$  and  $s_{t - \tau_{\mathrm{mix}} + 1}$ ,
$$
\left| \mathbb {E} \left[ F \left(\tilde {O} _ {t}, w _ {t - \tau_ {\operatorname {m i x}}}, \theta_ {t - \tau_ {\operatorname {m i x}}}\right) \right] \right| \leq 2 \sigma_ {w} U _ {f} C \beta^ {\tau_ {\operatorname {m i x}} - 1}.
$$
The following lemma, which reveals a useful property for critic update, is inspired by Olshovsky & Gharesifard (2022).
Lemma C.18. For critic update, we have the following two results:
$$
\begin{array}{l} \mathbb {E} \left[ \left(\hat {w} _ {\theta_ {t}} ^ {*} - \hat {w} _ {\theta_ {t + 1}} ^ {*}\right) ^ {T} \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} \left(w _ {t}, \theta_ {t}\right)\right) | \mathcal {F} _ {t} \right] \\ \leq \left(\sqrt {2} \alpha^ {\theta} L _ {w} | | \bar {g} (w _ {t}, \theta_ {t}) | | + \frac {\alpha^ {\theta^ {2}} H _ {w} U _ {g} ^ {2}}{\sqrt {2} (1 - \gamma) ^ {2}}\right) \cdot \left\| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} (w _ {t}, \theta_ {t}) \right\|, \\ \end{array}
$$
$$
\mathbb {E} \left[ \left(\hat {w} _ {\theta_ {t}} ^ {*} - \hat {w} _ {\theta_ {t + 1}} ^ {*}\right) ^ {T} \alpha^ {w} \left[ f (O _ {t}, w _ {t}) - \bar {f} (w _ {t}, \theta_ {t}) \right] \right] \leq 2 \alpha^ {w} U _ {f} \left(\frac {\sqrt {2} \alpha^ {\theta} L _ {w} U _ {g}}{1 - \gamma} + \frac {\alpha^ {\theta^ {2}} H _ {w} U _ {g} ^ {2}}{\sqrt {2} (1 - \gamma) ^ {2}}\right).
$$
Proof. If we use  $x(i)$  to denote the  $i$ 'th entry of vector  $x$ ,
$$
\left(\hat {w} _ {\theta_ {t}} ^ {*} - \hat {w} _ {\theta_ {t + 1}} ^ {*}\right) ^ {T} \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} \left(w _ {t}, \theta_ {t}\right)\right) = \sum_ {i} \left(\hat {w} _ {\theta_ {t}} ^ {*} (i) - \hat {w} _ {\theta_ {t + 1}} ^ {*} (i)\right) \cdot \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} \left(w _ {t}, \theta_ {t}\right)\right) (i).
$$
We can view  $\hat{w}_{\theta}^{*}$  as a function of  $\theta$ . Using a second order expansion,
If we take expectation conditioned on  $\mathcal{F}_t$
$$
\begin{array}{l} \mathbb {E} \left[ \hat {w} _ {\theta_ {t}} ^ {*} (i) - \hat {w} _ {\theta_ {t + 1}} ^ {*} (i) | \mathcal {F} _ {t} \right] = \mathbb {E} \left[ \nabla \hat {w} _ {\theta_ {t}} ^ {*} (i) ^ {T} (\theta_ {t} - \theta_ {t + 1}) | \mathcal {F} _ {t} \right] - \mathbb {E} \left[ \frac {1}{2} (\theta_ {t + 1} - \theta_ {t}) ^ {T} \nabla^ {2} \hat {w} _ {\theta_ {t} ^ {\prime}} ^ {*} (i) (\theta_ {t + 1} - \theta_ {t}) | \mathcal {F} _ {t} \right] \\ = \alpha^ {\theta} \nabla \hat {w} _ {\theta_ {t}} ^ {*} (i) ^ {T} \bar {g} (w _ {t}, \theta_ {t}) - \frac {\alpha^ {\theta^ {2}}}{2 (1 - \gamma) ^ {2}} \mathbb {E} \left[ g (\hat {O} _ {t}, w _ {t}, \theta_ {t}) ^ {T} \nabla^ {2} \hat {w} _ {\theta_ {t} ^ {\prime}} ^ {*} (i) g (\hat {O} _ {t}, w _ {t}, \theta_ {t}) | \mathcal {F} _ {t} \right]. \\ \end{array}
$$
This leads to
$$
\begin{array}{l} \mathbb {E} \left[ \hat {w} _ {\theta_ {t}} ^ {*} (i) - \hat {w} _ {\theta_ {t + 1}} ^ {*} (i) | \mathcal {F} _ {t} \right] ^ {2} \\ \leq 2 \alpha^ {\theta^ {2}} \left(\nabla \hat {w} _ {\theta_ {t}} ^ {*} (i) ^ {T} \bar {g} (w _ {t}, \theta_ {t})\right) ^ {2} + \frac {\alpha^ {\theta^ {4}}}{2 (1 - \gamma) ^ {4}} \mathbb {E} \left[ g (\hat {O} _ {t}, w _ {t}, \theta_ {t}) ^ {T} \nabla^ {2} \hat {w} _ {\theta_ {t} ^ {\prime}} ^ {*} (i) g (\hat {O} _ {t}, w _ {t}, \theta_ {t}) | \mathcal {F} _ {t} \right] ^ {2} \\ \leq 2 \alpha^ {\theta^ {2}} L _ {w} (i) ^ {2} | | \bar {g} (w _ {t}, \theta_ {t}) | | ^ {2} + \frac {\alpha^ {\theta^ {4}} H _ {w} (i) ^ {2} U _ {g} ^ {4}}{2 (1 - \gamma) ^ {4}}. \\ \end{array}
$$
where in the last line we use Assumption 2.6 and Lemma C.13. Now we go back to what we really care about,
$$
\begin{array}{l} \mathbb {E} \left[ \left(\hat {w} _ {\theta_ {t}} ^ {*} - \hat {w} _ {\theta_ {t + 1}} ^ {*}\right) ^ {T} \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} \left(w _ {t}, \theta_ {t}\right)\right) | \mathcal {F} _ {t} \right] \\ = \sum_ {i} \mathbb {E} \left[ \left(\hat {w} _ {\theta_ {t}} ^ {*} (i) - \hat {w} _ {\theta_ {t + 1}} ^ {*} (i)\right) \cdot \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} \left(w _ {t}, \theta_ {t}\right)\right) (i) | \mathcal {F} _ {t} \right] \\ = \sum_ {i} \mathbb {E} \left[ \hat {w} _ {\theta_ {t}} ^ {*} (i) - \hat {w} _ {\theta_ {t + 1}} ^ {*} (i) | \mathcal {F} _ {t} \right] \cdot \left[ \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} \left(w _ {t}, \theta_ {t}\right)\right) (i) \right] \\ \leq \sqrt {\sum_ {i} \mathbb {E} \left[ \hat {w} _ {\theta_ {t}} ^ {*} (i) - \hat {w} _ {\theta_ {t + 1}} ^ {*} (i) | \mathcal {F} _ {t} \right] ^ {2}} \cdot \sqrt {\sum_ {i} \left[ \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} \left(w _ {t} , \theta_ {t}\right)\right) (i) \right] ^ {2}} \\ \leq \sqrt {\sum_ {i} \left(2 \alpha^ {\theta^ {2}} L _ {w} (i) ^ {2} | | \bar {g} (w _ {t} , \theta_ {t}) | | ^ {2} + \frac {\alpha^ {\theta^ {4}} H _ {w} (i) ^ {2} U _ {g} ^ {4}}{2 (1 - \gamma) ^ {4}}\right)} \cdot \left\| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} (w _ {t}, \theta_ {t}) \right\| \\ \leq \left(\sqrt {2} \alpha^ {\theta} L _ {w} | | \bar {g} (w _ {t}, \theta_ {t}) | | + \frac {\alpha^ {\theta^ {2}} H _ {w} U _ {g} ^ {2}}{\sqrt {2} (1 - \gamma) ^ {2}}\right) \cdot \left| | w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} (w _ {t}, \theta_ {t}) \right|, \\ \end{array}
$$
where the third line is because the second term is constant conditioned on  $\mathcal{F}_t$ .
For the second part, notice that we already have the following result
$$
\begin{array}{l} \mathbb {E} \left[ \hat {w} _ {\theta_ {t}} ^ {*} (i) - \hat {w} _ {\theta_ {t + 1}} ^ {*} (i) | \mathcal {F} _ {t} \right] ^ {2} \leq 2 \alpha^ {\theta^ {2}} L _ {w} (i) ^ {2} | | \bar {g} (w _ {t}, \theta_ {t}) | | ^ {2} + \frac {\alpha^ {\theta^ {4}} H _ {w} (i) ^ {2} U _ {g} ^ {4}}{2 (1 - \gamma) ^ {4}} \\ \leq \frac {2 \alpha^ {\theta^ {2}} L _ {w} (i) ^ {2} U _ {g} ^ {2}}{(1 - \gamma) ^ {2}} + \frac {\alpha^ {\theta^ {4}} H _ {w} (i) ^ {2} U _ {g} ^ {4}}{2 (1 - \gamma) ^ {4}}, \\ \end{array}
$$
where we simply bound  $||\bar{g} (w_t,\theta_t)||$  by  $U_{g}$  (this result is from Lemma C.13). On the other hand, by Lemma C.15 and Lemma C.16, a rough bound for  $f(O_{t},w_{t}) - \bar{f} (w_{t},\theta_{t})$  would be simply
$$
\left\| f \left(O _ {t}, w _ {t}\right) - \bar {f} \left(w _ {t}, \theta_ {t}\right) \right\| \leq 2 U _ {f}.
$$
Now we go back to what we really care about,
$$
\begin{array}{l} \mathbb {E} \left[ \left(\hat {w} _ {\theta_ {t}} ^ {*} - \hat {w} _ {\theta_ {t + 1}} ^ {*}\right) ^ {T} \alpha^ {w} \left[ f (O _ {t}, w _ {t}) - \bar {f} (w _ {t}, \theta_ {t}) \right] | \mathcal {F} _ {t} \right] \\ = \alpha^ {w} \sum_ {i} \mathbb {E} \left[ \left(\hat {w} _ {\theta_ {t}} ^ {*} (i) - \hat {w} _ {\theta_ {t + 1}} ^ {*} (i)\right) \cdot \left(f \left(O _ {t}, w _ {t}\right) - \bar {f} \left(w _ {t}, \theta_ {t}\right)\right) (i) | \mathcal {F} _ {t} \right] \\ = \alpha^ {w} \sum_ {i} \mathbb {E} \left[ \hat {w} _ {\theta_ {t}} ^ {*} (i) - \hat {w} _ {\theta_ {t + 1}} ^ {*} (i) | \mathcal {F} _ {t} \right] \cdot \mathbb {E} \left[ \left(f \left(O _ {t}, w _ {t}\right) - \bar {f} \left(w _ {t}, \theta_ {t}\right)\right) (i) | \mathcal {F} _ {t} \right] \\ \leq \alpha^ {w} \sqrt {\sum_ {i} \mathbb {E} \left[ \hat {w} _ {\theta_ {t}} ^ {*} (i) - \hat {w} _ {\theta_ {t + 1}} ^ {*} (i) | \mathcal {F} _ {t} \right] ^ {2}} \cdot \sqrt {\sum_ {i} \mathbb {E} \left[ \left(f (O _ {t} , w _ {t}) - \bar {f} (w _ {t} , \theta_ {t})\right) (i) | \mathcal {F} _ {t} \right] ^ {2}} \\ \leq \alpha^ {w} \sqrt {\sum_ {i} \left(\frac {2 \alpha^ {\theta^ {2}} L _ {w} (i) ^ {2} U _ {g} ^ {2}}{(1 - \gamma) ^ {2}} + \frac {\alpha^ {\theta^ {4}} H _ {w} (i) ^ {2} U _ {g} ^ {4}}{2 (1 - \gamma) ^ {4}}\right)} \cdot \sqrt {\mathbb {E} \left[ \sum_ {i} \left(f (O _ {t} , w _ {t}) - \bar {f} (w _ {t} , \theta_ {t})\right) (i) ^ {2} | \mathcal {F} _ {t} \right]} \\ \leq 2 \alpha^ {w} U _ {f} \left(\frac {\sqrt {2} \alpha^ {\theta} L _ {w} U _ {g}}{1 - \gamma} + \frac {\alpha^ {\theta^ {2}} H _ {w} U _ {g} ^ {2}}{\sqrt {2} (1 - \gamma) ^ {2}}\right), \\ \end{array}
$$
where the third line is because the two terms are independent when conditioned on  $\mathcal{F}_t$ . The second part of this lemma is proved after we take expectation on both sides.
D Actor Update Analysis
Lemma D.1.
Proof. Actor update says the following:
$$
\begin{array}{l} \theta_ {t + 1} = \theta_ {t} - \frac {\alpha^ {\theta}}{1 - \gamma} g (\hat {O} _ {t}, w _ {t}, \theta_ {t}) \\ = \theta_ {t} - \alpha^ {\theta} \bar {g} (w _ {t}, \theta_ {t}) + \alpha^ {\theta} \bar {g} (w _ {t}, \theta_ {t}) - \frac {\alpha^ {\theta}}{1 - \gamma} g (\hat {O} _ {t}, w _ {t}, \theta_ {t}). \\ \end{array}
$$
By the definition of  $\bar{g} (w_t,\theta_t)$  in Eq.(16),
$$
\begin{array}{l} \bar {g} \left(w _ {t}, \theta_ {t}\right) = \nabla \ln \pi \left(\theta_ {t}\right) ^ {T} \Phi_ {\theta_ {t}} Q \left(w _ {t}\right) \\ = \nabla \ln \pi (\theta_ {t}) ^ {T} \Phi_ {\theta_ {t}} \left[ Q _ {\theta_ {t}} ^ {*} + Q (w _ {t}) - Q _ {\theta_ {t}} ^ {*} \right] \\ = \nabla V _ {\theta_ {t}} ^ {* T} + \nabla \ln \pi \left(\theta_ {t}\right) ^ {T} \Phi_ {\theta_ {t}} \left[ Q (w _ {t}) - Q _ {\theta_ {t}} ^ {*} \right], \\ \end{array}
$$
where we use the fact in Eq.(7).
For simplicity, denote  $D_{Q} = \nabla \ln \pi (\theta_{t})^{T}\Phi_{\theta_{t}}\left[Q(w_{t}) - Q_{\theta_{t}}^{*}\right]$ . So far we have the following result:
$$
\theta_ {t + 1} - \theta_ {t} = - \alpha^ {\theta} \nabla V _ {\theta_ {t}} ^ {*} - \alpha^ {\theta} D _ {Q} + \alpha^ {\theta} \bar {g} (w _ {t}, \theta_ {t}) - \frac {\alpha^ {\theta}}{1 - \gamma} g (\hat {O} _ {t}, w _ {t}, \theta_ {t}).
$$
On one hand, if we take expectation (conditioned on  $\mathcal{F}_t$ ) on both sides, we get
where we use the fact in Eq.(16) that  $\mathbb{E}\left[\frac{1}{1 - \gamma} g(\hat{O}_t, w_t, \theta_t)\right] = \bar{g}(w_t, \theta_t)$ . On the other hand,