SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \langle \boldsymbol {z} _ {t}, \bar {g} (\boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}) \rangle = \langle \boldsymbol {z} _ {t}, \bar {g} (\boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}) - \bar {g} (\boldsymbol {\omega} _ {t} ^ {*}, \boldsymbol {\theta} _ {t}) \rangle \\ = \left\langle z _ {t}, \mathbb {E} \left[ \left(\phi \left(s ^ {\prime}\right) - \phi (s)\right) ^ {\top} \left(\omega_ {t} - \omega_ {t} ^ {*}\right) \phi (s) \right] \right\rangle \\ = \boldsymbol {z} _ {t} ^ {\top} \mathbb {E} [ \phi (s) (\phi (s ^ {\prime}) - \phi (s)) ^ {\top} ] \boldsymbol {z} _ {t} \\ = z _ {t} ^ {\top} A z _ {t} \\ \leq - \lambda \| z _ {t} \| ^ {2}, \\ \end{array}
$$
where the first equality is due to the fact that  $\bar{g} (\omega_t^*,\pmb {\theta}_t) = 0$  and the last inequality follows from Assumption 3.1.
Taking expectation up to  $s_{t + 1}$ , we have
$$
\begin{array}{l} \mathbb {E} \| \boldsymbol {z} _ {t + 1} \| ^ {2} \leq \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} + 2 \beta_ {t} \mathbb {E} \langle \boldsymbol {z} _ {t}, \bar {g} (\boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}) \rangle + 2 \beta_ {t} \mathbb {E} \Psi (O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}) + 2 \beta_ {t} \mathbb {E} \langle \boldsymbol {z} _ {t}, \Delta g (O _ {t}, \eta_ {t}, \boldsymbol {\theta} _ {t}) \rangle \\ + 2 \mathbb {E} \left\langle \boldsymbol {z} _ {t}, \boldsymbol {\omega} _ {t} ^ {*} - \boldsymbol {\omega} _ {t + 1} ^ {*} \right\rangle + 2 U _ {\delta} ^ {2} \beta_ {t} ^ {2} + 2 \mathbb {E} \left\| \boldsymbol {\omega} _ {t} ^ {*} - \boldsymbol {\omega} _ {t + 1} ^ {*} \right\| ^ {2} \\ \leq \left(1 - 2 \lambda \beta_ {t}\right) \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} + 2 \beta_ {t} \mathbb {E} \Psi \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) + 2 \beta_ {t} \mathbb {E} \langle \boldsymbol {z} _ {t}, \Delta g (O _ {t}, \eta_ {t}, \boldsymbol {\theta} _ {t}) \rangle \\ + 2 \mathbb {E} \left\langle \boldsymbol {z} _ {t}, \boldsymbol {\omega} _ {t} ^ {*} - \boldsymbol {\omega} _ {t + 1} ^ {*} \right\rangle + 2 U _ {\delta} ^ {2} \beta_ {t} ^ {2} + 2 \mathbb {E} \left\| \boldsymbol {\omega} _ {t} ^ {*} - \boldsymbol {\omega} _ {t + 1} ^ {*} \right\| ^ {2} \\ \leq (1 - 2 \lambda \beta_ {t}) \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} + 2 \beta_ {t} \mathbb {E} \Psi \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) + 2 \beta_ {t} \mathbb {E} \langle \boldsymbol {z} _ {t}, \Delta g \left(O _ {t}, \eta_ {t}, \boldsymbol {\theta} _ {t}\right) \rangle \\ + 2 \mathbb {E} \left\langle z _ {t}, \boldsymbol {\omega} _ {t} ^ {*} - \boldsymbol {\omega} _ {t + 1} ^ {*} + \left(\nabla \boldsymbol {\omega} _ {t} ^ {*}\right) ^ {\top} \left(\boldsymbol {\theta} _ {t + 1} - \boldsymbol {\theta} _ {t}\right) \right\rangle \\ + 2 \mathbb {E} \langle \boldsymbol {z} _ {t}, (\nabla \boldsymbol {\omega} _ {t} ^ {*}) ^ {\top} (\boldsymbol {\theta} _ {t} - \boldsymbol {\theta} _ {t + 1}) \rangle + 2 U _ {\delta} ^ {2} \beta_ {t} ^ {2} + 2 \mathbb {E} \| \boldsymbol {\omega} _ {t} ^ {*} - \boldsymbol {\omega} _ {t + 1} ^ {*} \| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(1)} {\leq} (1 - 2 \lambda \beta_ {t}) \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} + 2 \beta_ {t} \mathbb {E} \Psi (O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}) + 2 \beta_ {t} \mathbb {E} \| \boldsymbol {z} _ {t} \| | y _ {t} | + L _ {s} \mathbb {E} \| \boldsymbol {z} _ {t} \| \| \boldsymbol {\theta} _ {t + 1} - \boldsymbol {\theta} _ {t} \| ^ {2} \\ + 2 \alpha_ {t} \mathbb {E} \langle z _ {t}, - (\nabla \omega_ {t} ^ {*}) ^ {\top} \delta_ {t} \nabla \log \pi_ {\boldsymbol {\theta} _ {t}} (a _ {t} | s _ {t}) \rangle + 2 U _ {\delta} ^ {2} \beta_ {t} ^ {2} + 2 L _ {*} ^ {2} \mathbb {E} \| \boldsymbol {\theta} _ {t} - \boldsymbol {\theta} _ {t + 1} \| ^ {2} \\ \leq \left(1 - 2 \lambda \beta_ {t}\right) \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} + 2 \beta_ {t} \mathbb {E} \Psi \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) + 2 \beta_ {t} \sqrt {\mathbb {E} y _ {t} ^ {2}} \sqrt {\mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}} \\ + \frac {L _ {s}}{2} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} \| \boldsymbol {\theta} _ {t + 1} - \boldsymbol {\theta} _ {t} \| ^ {2} + \frac {L _ {s}}{2} \mathbb {E} \| \boldsymbol {\theta} _ {t + 1} - \boldsymbol {\theta} _ {t} \| ^ {2} + 2 U _ {\delta} ^ {2} \beta_ {t} ^ {2} + 2 L _ {*} ^ {2} G ^ {2} \alpha_ {t} ^ {2} \\ + 2 \alpha_ {t} \mathbb {E} \left\langle \boldsymbol {z} _ {t}, - \left(\nabla \boldsymbol {\omega} _ {t} ^ {*}\right) ^ {\top} \delta_ {t} \nabla \log \pi_ {\boldsymbol {\theta} _ {t}} \left(a _ {t} | s _ {t}\right) \right\rangle \\ \leq (1 - 2 \lambda \beta_ {t}) \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} + 2 \beta_ {t} \mathbb {E} \Psi (O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}) + 2 \beta_ {t} \sqrt {\mathbb {E} y _ {t} ^ {2}} \sqrt {\mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}} + \frac {L _ {s} G ^ {2}}{2} \alpha_ {t} ^ {2} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} \\ + 2 U _ {\delta} ^ {2} \beta_ {t} ^ {2} + \left(2 L _ {*} ^ {2} + \frac {L _ {s}}{2}\right) G ^ {2} \alpha_ {t} ^ {2} + 2 \alpha_ {t} \mathbb {E} \left\langle z _ {t}, - \left(\nabla \omega_ {t} ^ {*}\right) ^ {\top} \delta_ {t} \nabla \log \pi_ {\boldsymbol {\theta} _ {t}} \left(a _ {t} \mid s _ {t}\right) \right\rangle \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(2)} {\leq} (1 - \lambda \beta_ {t}) \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} + 2 \beta_ {t} \mathbb {E} \Psi (O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}) + 2 \beta_ {t} \sqrt {\mathbb {E} y _ {t} ^ {2}} \sqrt {\mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}} \\ + 2 U _ {\delta} ^ {2} \beta_ {t} ^ {2} + \left(2 L _ {*} ^ {2} + \frac {L _ {s}}{2}\right) G ^ {2} \alpha_ {t} ^ {2} + 2 \alpha_ {t} \mathbb {E} \left\langle \boldsymbol {z} _ {t}, - \left(\nabla \omega_ {t} ^ {*}\right) ^ {\top} \delta_ {t} \nabla \log \pi_ {\boldsymbol {\theta} _ {t}} \left(a _ {t} \mid s _ {t}\right) \right\rangle \tag {19} \\ \end{array}
$$
where (1) follows from the  $L_{s}$ -smoothness of  $\omega^{*}$  in Lemma B.4; (2) uses  $\frac{L_sG^2}{2}\alpha_t^2\leq \lambda \beta_t$  for large  $T$ . For term  $\mathbb{E}\langle \pmb {z}_t, - (\nabla \pmb {\omega}_t^*)^\top \delta_t\nabla \log \pi_{\pmb{\theta}_t}(a_t|s_t)\rangle$ , we have
$$
\begin{array}{l} \mathbb {E} \left\langle \boldsymbol {z} _ {t}, - \left(\nabla \omega_ {t} ^ {*}\right) ^ {\top} \delta_ {t} \nabla \log \pi_ {\boldsymbol {\theta} _ {t}} \left(a _ {t} | s _ {t}\right) \right\rangle \\ = \mathbb {E} \left\langle \boldsymbol {z} _ {t}, \left(\nabla \omega_ {t} ^ {*}\right) ^ {\top} \left(- \Delta h \left(O _ {t}, \eta_ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) - h \left(O _ {t}, \boldsymbol {\theta} _ {t}\right)\right) \right\rangle \\ = - \mathbb {E} \left\langle \boldsymbol {z} _ {t}, \left(\nabla \omega_ {t} ^ {*}\right) ^ {\top} \Delta h \left(O _ {t}, \eta_ {t}, \omega_ {t}, \boldsymbol {\theta} _ {t}\right) \right\rangle \\ + \mathbb {E} \langle \boldsymbol {z} _ {t}, (\nabla \boldsymbol {\omega} _ {t} ^ {*}) ^ {\top} (\mathbb {E} _ {O _ {t} ^ {\prime}} [ h (O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t}) ] - h (O _ {t}, \boldsymbol {\theta} _ {t}) - \mathbb {E} _ {O _ {t} ^ {\prime}} [ h (O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t}) ]) \rangle \\ = \mathbb {E} \left[ \Xi \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) \right] - \mathbb {E} \left\langle \boldsymbol {z} _ {t}, \left(\nabla \boldsymbol {\omega} _ {t} ^ {*}\right) ^ {\top} \Delta h \left(O _ {t}, \eta_ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) \right\rangle - \mathbb {E} \left\langle \boldsymbol {z} _ {t}, \left(\nabla \boldsymbol {\omega} _ {t} ^ {*}\right) ^ {\top} \mathbb {E} _ {O _ {t} ^ {\prime}} \left[ h \left(O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t}\right) \right] \right\rangle \tag {20} \\ \end{array}
$$
Note that from Cauchy-Schwartz inequality and  $L_{*}$  is the Lipschitz constant of  $\omega^{*}$  in Lemma B.3, we have
$$
- \mathbb {E} \left\langle \boldsymbol {z} _ {t}, \left(\nabla \omega_ {t} ^ {*}\right) ^ {\top} \Delta h \left(O _ {t}, \eta_ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) \right\rangle \leq B L _ {*} \sqrt {\mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}} \sqrt {2 \mathbb {E} y _ {t} ^ {2} + 8 \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}}. \tag {21}
$$
Furthermore, it holds that
$$
\begin{array}{l} \mathbb {E} _ {O ^ {\prime}} \| \nabla h ^ {\prime} (O, \boldsymbol {\theta}) \| ^ {2} = \mathbb {E} _ {O ^ {\prime}} \| ((\phi (s) ^ {\top} \boldsymbol {\omega} ^ {*} - V _ {\boldsymbol {\theta}} (s ^ {\prime})) - (\phi (s) ^ {\top} \boldsymbol {\omega} ^ {*} - V _ {\boldsymbol {\theta}} (s))) \nabla \log \pi_ {\boldsymbol {\theta}} (a | s) \| ^ {2} \\ \leq \mathbb {E} _ {O ^ {\prime}} \left[ B ^ {2} \left(\left(\phi (s ^ {\prime}) ^ {\top} \omega^ {*} - V _ {\theta} (s ^ {\prime})\right) - \left(\phi (s) ^ {\top} \omega^ {*} - V _ {\theta} (s)\right)\right) ^ {2} \right] \\ \leq \mathbb {E} _ {\mathcal {O} ^ {\prime}} \left[ 2 B ^ {2} \left(\phi \left(s ^ {\prime}\right) ^ {\top} \omega^ {*} - V _ {\theta} \left(s ^ {\prime}\right)\right) ^ {2} + \left(\phi (s) ^ {\top} \omega^ {*} - V _ {\theta} (s)\right) ^ {2} \right] \\ = 4 B ^ {2} \mathbb {E} _ {O ^ {\prime}} \left[ \left(\phi (s) ^ {\top} \omega^ {*} - V _ {\theta} (s)\right) ^ {2} \right] \\ = 4 B ^ {2} \epsilon_ {\mathrm {a p p}} ^ {2}. \\ \end{array}
$$
$$
\begin{array}{l} - \left\langle \boldsymbol {z} _ {t}, \left(\nabla \omega_ {t} ^ {*}\right) ^ {\top} \mathbb {E} _ {O ^ {\prime}} \left[ \Delta h ^ {\prime} \left(O ^ {\prime}, \boldsymbol {\theta} _ {t}\right) \right] \right\rangle \leq U _ {\delta} L _ {*} \sqrt {\| \mathbb {E} _ {O ^ {\prime}} \left[ \Delta h ^ {\prime} \left(O _ {t} , \boldsymbol {\theta} _ {t}\right) \right] \| ^ {2}} \\ \leq U _ {\delta} L _ {*} \sqrt {\mathbb {E} _ {O ^ {\prime}} \| \Delta h ^ {\prime} \left(O _ {t} , \boldsymbol {\theta} _ {t}\right) \| ^ {2}} \\ \leq 2 B U _ {\delta} L _ {*} \epsilon_ {\text {a p p}}. \tag {22} \\ \end{array}
$$
Substituting (21) and (22) into (20) yields
$$
\begin{array}{l} \mathbb {E} \left\langle \boldsymbol {z} _ {t}, - \left(\nabla \boldsymbol {\omega} _ {t} ^ {*}\right) ^ {\top} \delta_ {t} \nabla \log \pi_ {\boldsymbol {\theta} _ {t}} \left(a _ {t} \mid s _ {t}\right) \right\rangle \leq \mathbb {E} \Xi \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) + 2 B U _ {\delta} L _ {*} \epsilon_ {\text {a p p}} \tag {23} \\ + B L _ {*} \sqrt {\mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}} \sqrt {2 \mathbb {E} y _ {t} ^ {2} + 8 \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}}. \\ \end{array}
$$
Plugging (23) into (19), we have
$$
\begin{array}{l} \mathbb {E} \| \boldsymbol {z} _ {t + 1} \| ^ {2} \leq (1 - \lambda \beta_ {t}) \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} + 2 \beta_ {t} \mathbb {E} \Psi \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) + 2 \alpha_ {t} \mathbb {E} \Xi \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) \\ + 2 \beta_ {t} \sqrt {\mathbb {E} y _ {t} ^ {2}} \sqrt {\mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}} + 2 B L _ {*} \alpha_ {t} \sqrt {\mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}} \sqrt {2 \mathbb {E} y _ {t} ^ {2} + 8 \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}} \tag {24} \\ + 2 U _ {\delta} ^ {2} \beta_ {t} ^ {2} + (2 L _ {*} ^ {2} + \frac {L _ {s}}{2}) G ^ {2} \alpha_ {t} ^ {2} + 4 B U _ {\delta} \alpha_ {t} \epsilon_ {\mathrm {a p p}}. \\ \end{array}
$$
Rearranging and summing from  $\tau_T$  to  $T$  gives
$$
\begin{array}{l} \lambda \sum_ {\tau_ {T}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} \leq \underbrace {\sum_ {t = \tau_ {T}} ^ {T - 1} \frac {1}{\beta_ {t}} (\mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} - \mathbb {E} \| \boldsymbol {z} _ {t + 1} \| ^ {2})} _ {I _ {1}} + \underbrace {2 \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \Psi (O _ {t} , \boldsymbol {\omega} _ {t} , \boldsymbol {\theta} _ {t})} _ {I _ {2}} + \underbrace {2 c \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \Xi (O _ {t} , \boldsymbol {\omega} _ {t} , \boldsymbol {\theta} _ {t})} _ {I _ {3}} \\ + \underbrace {2 \sum_ {t = \tau_ {T}} ^ {T - 1} \sqrt {\mathbb {E} y _ {t} ^ {2}} \sqrt {\mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}}} _ {I _ {4}} + \underbrace {2 c B L _ {*} \sum_ {t = \tau_ {T}} ^ {T - 1} \sqrt {\mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}} \sqrt {2 \mathbb {E} y _ {t} ^ {2} + 8 \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}}} _ {I _ {5}} \\ + \sum_ {t = \tau_ {T}} ^ {T - 1} \big (2 U _ {\delta} ^ {2} \beta_ {t} + c (2 L _ {*} ^ {2} + \frac {L _ {s}}{2}) G ^ {2} \alpha_ {t} + 4 c B U _ {\delta} \epsilon_ {\mathrm {a p p}} \big). \\ \end{array}
$$
In the sequel, we will tackle  $I_1, I_2, I_3, I_4, I_5$  respectively.
$$
\begin{array}{l} I _ {1} = \sum_ {t = \tau_ {T}} ^ {T - 1} \frac {1}{\beta_ {t}} \left(\mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} - \mathbb {E} \| \boldsymbol {z} _ {t + 1} \| ^ {2}\right) \\ = \sum_ {t = \tau_ {T} + 1} ^ {T - 1} \left(\frac {1}{\beta_ {t}} - \frac {1}{\beta_ {t - 1}}\right) \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} + \frac {1}{\beta_ {\tau_ {T}}} \mathbb {E} \| \boldsymbol {z} _ {\tau_ {T}} \| ^ {2} - \frac {1}{\beta_ {T - 1}} \mathbb {E} \| \boldsymbol {z} _ {T} \| ^ {2} \\ \leq U _ {\delta} ^ {2} \left(\sum_ {t = \tau_ {T} + 1} ^ {T - 1} \left(\frac {1}{\beta_ {t}} - \frac {1}{\beta_ {t - 1}}\right) + \frac {1}{\beta_ {\tau_ {T}}}\right) \\ = U _ {\delta} ^ {2} \sqrt {T}, \\ \end{array}
$$
where the inequality is due to  $\mathbb{E}\| \pmb {z}_t\| ^2\leq U_\delta^2$  and discard the last term.
For term  $I_{2}$ , from Lemma C.3, choose  $\tau = \tau_{T}$ , we have
$$
\begin{array}{l} \mathbb {E} \Psi (O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}) \leq C _ {1} \| \boldsymbol {\theta} _ {t} - \boldsymbol {\theta} _ {t - \tau_ {T}} \| + U _ {\delta} ^ {2} | \mathcal {A} | L _ {\pi} G \tau_ {T} (\tau_ {T} + 1) \alpha_ {t - \tau_ {T}} \\ + 2 U _ {\delta} ^ {2} m \rho^ {\tau_ {T} - 1} + 6 U _ {\delta} \| \boldsymbol {\omega} _ {t} - \boldsymbol {\omega} _ {t - \tau_ {T}} \| \\ \leq C _ {1} \sum_ {k = t - \tau_ {T}} ^ {t - 1} G \alpha_ {k} + U _ {\delta} ^ {2} | \mathcal {A} | L _ {\pi} G \tau_ {T} (\tau_ {T} + 1) \alpha_ {t - \tau_ {T}} + \frac {2 U _ {\delta} ^ {2}}{\sqrt {T}} + 6 U _ {\delta} \sum_ {k = t - \tau_ {T}} ^ {t - 1} U _ {\delta} \beta_ {k} \\ \leq \left(C _ {1} G \tau_ {T} + U _ {\delta} ^ {2} | \mathcal {A} | L _ {\pi} G \tau_ {T} \left(\tau_ {T} + 1\right)\right) \alpha_ {t - \tau_ {T}} + \frac {2 U _ {\delta} ^ {2}}{\sqrt {T}} + 6 U _ {\delta} ^ {2} \tau_ {T} \beta_ {t - \tau_ {T}}. \\ \end{array}
$$
Then we get
$$
\begin{array}{l} I _ {2} = 2 \sum_ {T = \tau_ {T}} ^ {T - 1} \mathbb {E} \Psi \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) \\ \leq 2 \sum_ {T = \tau_ {T}} ^ {T - 1} \left(\left(C _ {1} G \tau_ {T} + U _ {\delta} ^ {2} | \mathcal {A} | L _ {\pi} G \tau_ {T} (\tau_ {T} + 1)\right) \alpha_ {t} + \frac {2 U _ {\delta} ^ {2}}{\sqrt {T}} + 6 U _ {\delta} ^ {2} \tau_ {T} \beta_ {t}\right). \\ \end{array}
$$
For term  $I_{3}$ , from Lemma C.4, choose  $\tau = \tau_{T}$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \Xi \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) \right] \leq C _ {2} \left\| \boldsymbol {\theta} _ {t} - \boldsymbol {\theta} _ {t - \tau} \right\| + 2 U _ {\delta} B \left\| \boldsymbol {\omega} _ {t} - \boldsymbol {\omega} _ {t - \tau} \right\| \\ + 2 U _ {\delta} ^ {2} B | \mathcal {A} | L _ {\pi} G \tau (\tau + 1) \alpha_ {t - \tau} + 4 U _ {\delta} ^ {2} B m \rho^ {- 1} \\ \leq C _ {2} \sum_ {k = t - \tau_ {T}} ^ {t - 1} G \alpha_ {k} + 2 U _ {\delta} B \sum_ {k = t - \tau_ {T}} ^ {t - 1} U _ {\delta} \beta_ {k} \\ + 2 U _ {\delta} ^ {2} B | A | L _ {\pi} G \tau (\tau + 1) \alpha_ {t - \tau} + 4 U _ {\delta} ^ {2} B m \rho^ {\tau_ {T} - 1} \\ \leq \left(C _ {2} G \tau_ {T} + 2 U _ {\delta} ^ {2} B | \mathcal {A} | L _ {\pi} G \tau_ {T} (\tau_ {T} + 1)\right) \alpha_ {t} + 2 U _ {\delta} ^ {2} B \tau_ {T} \beta_ {t} + \frac {4 U _ {\delta} ^ {2} B}{\sqrt {T}}. \\ \end{array}
$$
$$
\begin{array}{l} I _ {3} = 2 c \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \Xi \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) \\ \leq 2 c \sum_ {t = \tau_ {T}} ^ {T - 1} \left(\left(C _ {2} G \tau_ {T} + 2 U _ {\delta} ^ {2} B | \mathcal {A} | L _ {\pi} G \tau_ {T} (\tau_ {T} + 1)\right) \alpha_ {t} + 2 U _ {\delta} ^ {2} B \tau_ {T} \beta_ {t} + \frac {4 U _ {\delta} ^ {2} B}{\sqrt {T}}\right). \\ \end{array}
$$
For term  $I_4$  and  $I_5$ , from Cauchy-Schwartz inequality, we have
$$
\begin{array}{l} I _ {4} \leq 2 \left(\sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} y _ {t} ^ {2}\right) ^ {\frac {1}{2}} \left(\sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}\right) ^ {\frac {1}{2}}, \\ I _ {5} \leq 2 c B L _ {*} (\sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}) ^ {\frac {1}{2}} (2 \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} y _ {t} ^ {2} + 8 \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}) ^ {\frac {1}{2}}. \\ \end{array}
$$
$$
\begin{array}{l} \lambda \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} \leq 2 \left(\sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} y _ {t} ^ {2}\right) ^ {\frac {1}{2}} \left(\sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}\right) ^ {\frac {1}{2}} \\ + 2 c B L _ {*} (\sum_ {t = \tau_ {t}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}) ^ {\frac {1}{2}} (2 \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} y _ {t} ^ {2} + 8 \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}) ^ {\frac {1}{2}} \\ + U _ {\delta} ^ {2} \sqrt {T} + 2 \sum_ {T = \tau_ {T}} ^ {T - 1} \left(\left(C _ {1} G \tau_ {T} + U _ {\delta} ^ {2} | \mathcal {A} | L _ {\pi} G \tau_ {T} (\tau_ {T} + 1)\right) \alpha_ {t} + \frac {2 U _ {\delta} ^ {2}}{\sqrt {T}} + 6 U _ {\delta} ^ {2} \tau_ {T} \beta_ {t}\right) \\ + 2 c \sum_ {t = \tau_ {T}} ^ {T - 1} \left(\left(C _ {2} G \tau_ {T} + 2 U _ {\delta} ^ {2} B | \mathcal {A} | L _ {\pi} G \tau_ {T} (\tau_ {T} + 1)\right) \alpha_ {t} + 2 U _ {\delta} ^ {2} B \tau_ {T} \beta_ {t} + \frac {4 U _ {\delta} ^ {2} B}{\sqrt {T}}\right) \\ + \sum_ {t = \tau_ {T}} ^ {T - 1} (2 U _ {\delta} ^ {2} \beta_ {t} + c (2 L _ {*} ^ {2} + \frac {L _ {s}}{2}) G ^ {2} \alpha_ {t} + 4 c B U _ {\delta} \epsilon_ {\mathrm {a p p}}). \\ \end{array}
$$
$$
\begin{array}{l} Z _ {T} \stackrel {(1)} {\leq} \frac {2}{\lambda} (\frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} y _ {t} ^ {2}) ^ {\frac {1}{2}} (\frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}) ^ {\frac {1}{2}} \\ + \frac {2 c B L _ {*}}{\lambda} \left(\frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {t}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}\right) ^ {\frac {1}{2}} \left(2 \frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} y _ {t} ^ {2} + 8 \frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}\right) ^ {\frac {1}{2}} \\ + \frac {1}{\lambda} \left(\frac {2 U _ {\delta} ^ {2}}{\sqrt {T}} + 2 \left(C _ {1} G \tau_ {T} + U _ {\delta} ^ {2} \right| \mathcal {A} | L _ {\pi} G \tau_ {T} (\tau_ {T} + 1)\right) \alpha_ {t} + \frac {4 U _ {\delta} ^ {2}}{\sqrt {T}} + 1 2 U _ {\delta} ^ {2} \tau_ {T} \beta_ {t} \\ + 2 c \left(C _ {2} G \tau_ {T} + 2 U _ {\delta} ^ {2} B | \mathcal {A} | L _ {\pi} G \tau_ {T} (\tau_ {T} + 1)\right) \alpha_ {t} + 4 c U _ {\delta} ^ {2} B \tau_ {T} \beta_ {t} + \frac {8 c U _ {\delta} ^ {2} B}{\sqrt {T}} \\ + 2 U _ {\delta} ^ {2} \beta_ {t} + c \left(2 L _ {*} ^ {2} + \frac {L _ {s}}{2}\right) G ^ {2} \alpha_ {t} + 4 c B U _ {\delta} \epsilon_ {\mathrm {a p p}}) \\ = \mathcal {O} (\frac {\log^ {2} T}{\sqrt {T}}) + \mathcal {O} (\epsilon_ {\mathrm {a p p}}) + \frac {2}{\lambda} (\frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} y _ {t} ^ {2}) ^ {\frac {1}{2}} (\frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}) ^ {\frac {1}{2}} \\ + \frac {2 c B L _ {*}}{\lambda} (\frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {t}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}) ^ {\frac {1}{2}} (2 \frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} y _ {t} ^ {2} + 8 \frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2}) ^ {\frac {1}{2}}, \\ \end{array}
$$
where (1) follows from  $\tau_T = \mathcal{O}(\log T)$  so that  $T - \tau_T \geq \frac{1}{2} T$  for large  $T$ . Therefore, we have
$$
Z _ {T} \leq \mathcal {O} (\frac {\log^ {2} T}{\sqrt {T}}) + \mathcal {O} (\epsilon_ {\mathrm {a p p}}) + \frac {2}{\lambda} \sqrt {Y _ {T} Z _ {T}} + \frac {2 c B L _ {*}}{\lambda} \sqrt {Z _ {T} (2 Y _ {T} + 8 Z _ {T})},
$$
C.3 Step 3: Policy gradient norm analysis
In this subsection, we will establish an implicit upper bound for policy gradient norm.
Lemma C.6. For any  $t \geq \tau > 0$ , it holds that
$$
\mathbb {E} \left[ \Theta \left(O _ {t}, \boldsymbol {\theta} _ {t}\right) \right] \leq D _ {1} \tau (\tau + 1) G \alpha + D _ {2} m \rho^ {\tau - 1},
$$
where  $D_{1} = \max \{U_{\delta}BL_{J^{\prime}} + 3L_{J}L_{h},2U_{\delta}BL_{J}|A|L_{\pi}\}$  and  $D_{2} = 4U_{\delta}BL_{J}$ .
Theorem C.7. We have
$$
G _ {T} \leq \mathcal {O} \left(\frac {\log^ {2} T}{\sqrt {T}}\right) + \mathcal {O} \left(\epsilon_ {\mathrm {a p p}}\right) + B \sqrt {G _ {T} \left(2 Y _ {T} + 8 Z _ {T}\right)}. \tag {25}
$$
Proof. From the update rule of actor in Line 9 of Algorithm 1 and 15, we have