SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ | | \theta_ {t + 1} - \theta_ {t} | | ^ {2} | \mathcal {F} _ {t} \right] = | | \alpha^ {\theta} \nabla V _ {\theta_ {t}} ^ {*} + \alpha^ {\theta} D _ {Q} | | ^ {2} + \mathbb {E} \left[ | | \alpha^ {\theta} \bar {g} (w _ {t}, \theta_ {t}) - \frac {\alpha^ {\theta}}{1 - \gamma} g (\hat {O} _ {t}, w _ {t}, \theta_ {t}) | | ^ {2} | \mathcal {F} _ {t} \right] \\ \leq | | \alpha^ {\theta} \nabla V _ {\theta_ {t}} ^ {*} + \alpha^ {\theta} D _ {Q} | | ^ {2} + \frac {4}{(1 - \gamma) ^ {2}} \alpha^ {\theta^ {2}} U _ {g} ^ {2}. \\ \end{array}
$$
where we keep using  $\mathbb{E}\left[\frac{1}{1 - \gamma} g(\hat{O}_t, w_t, \theta_t)\right] = \bar{g}(w_t, \theta_t)$ . Using Lemma C.12,
$$
V _ {\theta_ {t + 1}} ^ {*} \leq V _ {\theta_ {t}} ^ {*} + \nabla V _ {\theta_ {t}} ^ {*} (\theta_ {t + 1} - \theta_ {t}) + \frac {H _ {V}}{2} | | \theta_ {t + 1} - \theta_ {t} | | ^ {2}.
$$
Taking expectation (conditioned on  $\mathcal{F}_t$ ) on both sides and we obtain
$$
\mathbb {E} \left[ V _ {\theta_ {t + 1}} ^ {*} | \mathcal {F} _ {t} \right] \leq V _ {\theta_ {t}} ^ {*} + \nabla V _ {\theta_ {t}} ^ {*} \cdot \mathbb {E} \left[ \theta_ {t + 1} - \theta_ {t} | \mathcal {F} _ {t} \right] + \frac {H _ {V}}{2} \mathbb {E} \left[ | | \theta_ {t + 1} - \theta_ {t} | | ^ {2} | \mathcal {F} _ {t} \right].
$$
Plug in the facts about  $\theta_{t + 1} - \theta_t$ , we know
$$
\mathbb {E} \left[ V _ {\theta_ {t + 1}} ^ {*} | \mathcal {F} _ {t} \right] \leq V _ {\theta_ {t}} ^ {*} - \alpha^ {\theta} \| \nabla V _ {\theta_ {t}} ^ {*} \| ^ {2} - \alpha^ {\theta} \nabla V _ {\theta_ {t}} ^ {*} D _ {Q} + \frac {H _ {V}}{2} \| \alpha^ {\theta} \nabla V _ {\theta_ {t}} ^ {* T} + \alpha^ {\theta} D _ {Q} \| ^ {2} + \frac {2 H _ {V}}{(1 - \gamma) ^ {2}} \alpha^ {\theta^ {2}} U _ {g} ^ {2}.
$$
We can use the facts that  $2ab \leq a^2 + b^2$  and  $(a + b)^2 \leq 2a^2 + 2b^2$  to obtain
$$
\begin{array}{l} \mathbb {E} \left[ V _ {\theta_ {t + 1}} ^ {*} | \mathcal {F} _ {t} \right] \leq V _ {\theta_ {t}} ^ {*} - \alpha^ {\theta} \| \nabla V _ {\theta_ {t}} ^ {*} \| ^ {2} + \frac {\alpha^ {\theta}}{2} \| \nabla V _ {\theta_ {t}} ^ {*} \| ^ {2} + \frac {\alpha^ {\theta}}{2} \| D _ {Q} \| ^ {2} \\ + \alpha^ {\theta^ {2}} H _ {V} | | \nabla V _ {\theta_ {t}} ^ {*} | | ^ {2} + \alpha^ {\theta^ {2}} H _ {V} | | D _ {Q} | | ^ {2} + \frac {2 H _ {V}}{(1 - \gamma) ^ {2}} \alpha^ {\theta^ {2}} U _ {g} ^ {2} \\ = V _ {\theta_ {t}} ^ {*} + \left(\alpha_ {\theta} ^ {2} H _ {V} - \frac {\alpha_ {\theta}}{2}\right) \left\| \nabla V _ {\theta_ {t}} ^ {*} \right\| ^ {2} + \left(\alpha_ {\theta} ^ {2} H _ {V} + \frac {\alpha_ {\theta}}{2}\right) \left\| D _ {Q} \right\| ^ {2} + \frac {2 H _ {V}}{(1 - \gamma) ^ {2}} \alpha^ {\theta^ {2}} U _ {g} ^ {2} \\ \leq V _ {\theta_ {t}} ^ {*} + \left(\alpha_ {\theta} ^ {2} H _ {V} - \frac {\alpha_ {\theta}}{2}\right) \left| \left| \nabla V _ {\theta_ {t}} ^ {*} \right| \right| ^ {2} + \frac {2 H _ {V}}{(1 - \gamma) ^ {2}} \alpha^ {\theta 2} U _ {g} ^ {2} \\ \left. + \left(\alpha_ {\theta} ^ {2} H _ {V} + \frac {\alpha_ {\theta}}{2}\right) \cdot \left[ c _ {1} \mathcal {N} _ {\theta_ {t}} \left(Q (w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*}\right) + c _ {2} \epsilon^ {2} \right], \right. \\ \end{array}
$$
where the last inequality we uses the fact from LemmaC.4. We can rewrite it as
$$
\begin{array}{l} \left(\frac {\alpha_ {\theta}}{2} - \alpha_ {\theta} ^ {2} H _ {V}\right) | | \nabla V _ {\theta_ {t}} ^ {*} | | ^ {2} \leq V _ {\theta_ {t}} ^ {*} - \mathbb {E} \left[ V _ {\theta_ {t + 1}} ^ {*} | \mathcal {F} _ {t} \right] + \frac {2 H _ {V}}{(1 - \gamma) ^ {2}} \alpha^ {\theta^ {2}} U _ {g} ^ {2} \\ + \left(\alpha_ {\theta} ^ {2} H _ {V} + \frac {\alpha_ {\theta}}{2}\right) \cdot \left[ c _ {1} \mathcal {N} _ {\theta_ {t}} (Q (w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*}) + c _ {2} \epsilon^ {2} \right]. \\ \end{array}
$$
Taking expectation on both sides and telescoping sum:
$$
\begin{array}{l} \left(\frac {\alpha_ {\theta}}{2} - \alpha_ {\theta} ^ {2} H _ {V}\right) \frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ | | \nabla V _ {\theta_ {t}} ^ {*} | | ^ {2} \right] \leq \frac {\mathbb {E} \left[ V _ {\theta_ {1}} ^ {*} - V _ {\theta_ {T + 1}} ^ {*} \right]}{T} + c _ {2} \left(\alpha_ {\theta} ^ {2} H _ {V} + \frac {\alpha_ {\theta}}{2}\right) \epsilon^ {2} + \frac {2 H _ {V}}{(1 - \gamma) ^ {2}} \alpha^ {\theta^ {2}} U _ {g} ^ {2} \\ + c _ {1} \left(\alpha_ {\theta} ^ {2} H _ {V} + \frac {\alpha_ {\theta}}{2}\right) \frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathcal {N} _ {\theta_ {t}} (Q (w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*}) \right]. \\ \end{array}
$$
If we use notations from Eq.(13), the above fact can be rewritten as
E Critic Update Analysis under the Markov Sampling Case
Lemma E.1. In the Markov sampling case,
$$
\begin{array}{l} \left(2 \alpha^ {w} - \frac {1}{\sqrt {2}} \alpha^ {\theta} L _ {w} \left(\frac {1}{\lambda_ {\min }} - 2 \alpha^ {w}\right) - \sqrt {2} \alpha^ {\theta} L _ {w} \sqrt {c _ {1} \left(\frac {1}{\lambda_ {\min }} - 2 \alpha^ {w}\right)} - 2 \alpha^ {\theta 2} L _ {w} ^ {2} c _ {1}\right) \cdot \Delta_ {Q} \\ \leq \frac {\mathbb {E} \left[ | | w _ {1} - \hat {w} _ {\theta_ {1}} ^ {*} | | ^ {2} \right]}{T} + \left(\frac {1}{\sqrt {2}} \alpha^ {\theta} L _ {w} + 2 \alpha^ {\theta^ {2}} L _ {w} ^ {2}\right) \Delta_ {V} \\ + \left(\sqrt {2} \alpha^ {\theta} L _ {w} \sqrt {c _ {1} C _ {w , 1} + c _ {2} \epsilon^ {2} \left(\frac {1}{\lambda_ {\min}} - 2 \alpha^ {w}\right)} + \frac {\alpha^ {\theta^ {2}} H _ {w} U _ {g} ^ {2}}{\sqrt {2} (1 - \gamma) ^ {2}} \sqrt {\frac {1}{\lambda_ {\min}} - 2 \alpha^ {w}}\right) \sqrt {\Delta_ {Q}} \\ + C _ {w, 1} + C _ {w, 2} + \frac {1}{\sqrt {2}} \alpha^ {\theta} L _ {w} C _ {w, 1} + \sqrt {2} \alpha^ {\theta} L _ {w} \sqrt {c _ {2} C _ {w , 1} \epsilon^ {2}} + \frac {\sqrt {C _ {w , 1}} \alpha^ {\theta^ {2}} H _ {w} U _ {g} ^ {2}}{\sqrt {2} (1 - \gamma) ^ {2}} \\ + 2 \alpha^ {w} U _ {f} \left(\frac {\sqrt {2} \alpha^ {\theta} L _ {w} U _ {g}}{1 - \gamma} + \frac {\alpha^ {\theta^ {2}} H _ {w} U _ {g} ^ {2}}{\sqrt {2} (1 - \gamma) ^ {2}}\right) + 2 \alpha^ {\theta^ {2}} L _ {w} ^ {2} c _ {2} \epsilon^ {2} + \frac {4 L _ {w} ^ {2}}{(1 - \gamma) ^ {2}} \alpha^ {\theta^ {2}} U _ {g} ^ {2}, \\ \end{array}
$$
where, for simplicity, we denote
$$
C _ {w, 1} = 1 6 \alpha^ {w} (1 + \gamma) L _ {Q} H _ {Q} \sigma_ {w} ^ {3} + 4 \alpha^ {w} (1 + \gamma) L _ {Q} \epsilon \sigma_ {w} + \alpha^ {w 2} U _ {f} ^ {2},
$$
$$
C _ {w, 2} = 2 \alpha^ {w} \alpha^ {\theta} \tau_ {\text {m i x}} \left(L _ {F _ {\theta}} + 4 \sigma_ {w} U _ {f} n _ {a} L _ {\pi} \left(\tau_ {\text {m i x}} - 1\right)\right) U _ {g} + 2 \alpha^ {w ^ {2}} \tau_ {\text {m i x}} L _ {F _ {w}} \sigma_ {w} + 4 \alpha^ {w} \sigma_ {w} U _ {f} C \beta^ {\tau_ {\text {m i x}} - 1}.
$$
Proof. Recall the critic update is
$$
w _ {t + 1} = \operatorname {P r o j} _ {W} \left\{w _ {t} + \alpha^ {w} f \left(O _ {t}, w _ {t}\right) \right\},
$$
$$
\begin{array}{l} \left\| w _ {t + 1} - \hat {w} _ {\theta_ {t + 1}} ^ {*} \right\| ^ {2} = \left\| \mathbf {P r o j} _ {W} \left\{w _ {t} + \alpha^ {w} f \left(O _ {t}, w _ {t}\right) \right\} - \hat {w} _ {\theta_ {t + 1}} ^ {*} \right\| ^ {2} \\ \leq \left\| w _ {t} + \alpha^ {w} f \left(O _ {t}, w _ {t}\right) - \hat {w} _ {\theta_ {t + 1}} ^ {*} \right\| ^ {2} \\ = \left\| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \hat {w} _ {\theta_ {t}} ^ {*} - \hat {w} _ {\theta_ {t + 1}} ^ {*} + \alpha^ {w} f \left(O _ {t}, w _ {t}\right) \right\| ^ {2} \\ \leq \left\| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} f \left(O _ {t}, w _ {t}\right) \right\| ^ {2} \\ + 2 \left(\hat {w} _ {\theta_ {t} ^ {*}} - \hat {w} _ {\theta_ {t + 1} ^ {*}}\right) ^ {T} \left(w _ {t} - \hat {w} _ {\theta_ {t} ^ {*}} + \alpha^ {w} f \left(O _ {t}, w _ {t}\right)\right) + \left\| \hat {w} _ {\theta_ {t} ^ {*}} - \hat {w} _ {\theta_ {t + 1} ^ {*}} ^ {*} \right\| ^ {2}. \\ \end{array}
$$
We can take expectation on both sides,
$$
\begin{array}{l} \mathbb {E} \left[ \left| \left| w _ {t + 1} - \hat {w} _ {\theta_ {t + 1}} ^ {*} \right| \right| ^ {2} \right] \\ \leq \underbrace {\mathbb {E} \left[ \left| \left| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} f \left(O _ {t} , w _ {t}\right) \right| \right| ^ {2} \right]} _ {I _ {1}} \tag {17} \\ + \underbrace {\mathbb {E} \left[ 2 \left(\hat {w} _ {\theta_ {t}} ^ {*} - \hat {w} _ {\theta_ {t + 1}} ^ {*}\right) ^ {T} \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} f (O _ {t} , w _ {t})\right) \right]} _ {I _ {2}} + \underbrace {\mathbb {E} \left[ \left| | \hat {w} _ {\theta_ {t}} ^ {*} - \hat {w} _ {\theta_ {t + 1}} ^ {*} \right| ^ {2} \right]} _ {I _ {3}}. \\ \end{array}
$$
To analyze  $I_{1}$ , we derive
$$
\begin{array}{l} \mathbb {E} \left[ | | w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} f (O _ {t}, w _ {t}) | | ^ {2} \right] \\ = \mathbb {E} \left[ \left| \left| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} \right| \right| ^ {2} + 2 \alpha^ {w} \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}\right) ^ {T} f (O _ {t}, w _ {t}) + \alpha^ {w 2} \left| \left| f (O _ {t}, w _ {t}) \right| \right| ^ {2} \right] \\ = \mathbb {E} \left[ \left\| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} \right\| ^ {2} \right] + \underbrace {\mathbb {E} \left[ 2 \alpha^ {w} \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}\right) ^ {T} \bar {f} \left(w _ {t} , \theta_ {t}\right) \right]} _ {I _ {1, 1}} \tag {18} \\ + \underbrace {\mathbb {E} \left[ \alpha^ {w 2} | | f (O _ {t} , w _ {t}) | | ^ {2} \right]} _ {I _ {1, 2}} + \underbrace {\mathbb {E} \left[ 2 \alpha^ {w} (w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}) ^ {T} \left[ f (O _ {t} , w _ {t}) - \bar {f} (w _ {t} , \theta_ {t}) \right] \right]} _ {I _ {1, 3}}. \\ \end{array}
$$
To analyze  $I_{1,1}$ , we perform the following of manipulations:
$$
\begin{array}{l} 2 \alpha^ {w} \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}\right) ^ {T} \bar {f} \left(w _ {t}, \theta_ {t}\right) \\ = 2 \alpha^ {w} \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}\right) ^ {T} \nabla Q \left(w _ {t}\right) ^ {T} D _ {\theta_ {t}} \left(\gamma P _ {\theta_ {t}} - I\right) \left(Q \left(w _ {t}\right) - Q _ {\theta_ {t}} ^ {*}\right) \\ = 2 \alpha^ {w} \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}\right) ^ {T} \nabla Q \left(w _ {t}\right) ^ {T} D _ {\theta_ {t}} \left(\gamma P _ {\theta_ {t}} - I\right) \left(Q \left(w _ {t}\right) - \hat {Q} _ {\theta_ {t}} ^ {*} + \hat {Q} _ {\theta_ {t}} ^ {*} - Q _ {\theta_ {t}} ^ {*}\right) \\ = 2 \alpha^ {w} \left[ \left(Q (w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*}\right) ^ {T} + \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}\right) ^ {T} \nabla Q (w _ {t}) ^ {T} - \left(Q (w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*}\right) ^ {T} \right] D _ {\theta_ {t}} (\gamma P _ {\theta_ {t}} - I) \left(Q (w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*}\right) \\ + 2 \alpha^ {w} \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}\right) ^ {T} \nabla Q \left(w _ {t}\right) ^ {T} D _ {\theta_ {t}} \left(\gamma P _ {\theta_ {t}} - I\right) \left(\hat {Q} _ {\theta_ {t}} ^ {*} - Q _ {\theta_ {t}} ^ {*}\right) \\ = - 2 \alpha^ {w} \mathcal {N} _ {\theta_ {t}} (Q (w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*}) + 2 \alpha^ {w} (w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}) ^ {T} (\nabla Q (w _ {t}) - \nabla Q (w _ {\text {m i d}})) ^ {T} D _ {\theta_ {t}} (\gamma P _ {\theta_ {t}} - I) (Q (w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*}) \\ + 2 \alpha^ {w} \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}\right) ^ {T} \nabla Q \left(w _ {t}\right) ^ {T} D _ {\theta_ {t}} \left(\gamma P _ {\theta_ {t}} - I\right) \left(\hat {Q} _ {\theta_ {t}} ^ {*} - Q _ {\theta_ {t}} ^ {*}\right) \\ \leq - 2 \alpha^ {w} \mathcal {N} _ {\theta_ {t}} (Q (w _ {t}) - \hat {Q} _ {\theta_ {t} ^ {*}}) + 2 \alpha^ {w} (1 + \gamma) L _ {Q} H _ {Q} \| w _ {t} - \hat {w} _ {\theta_ {t} ^ {*}} ^ {*} \| ^ {3} + 2 \alpha^ {w} (1 + \gamma) L _ {Q} \epsilon \| w _ {t} - \hat {w} _ {\theta_ {t} ^ {*}} ^ {*} \| \\ \leq - 2 \alpha^ {w} \mathcal {N} _ {\theta_ {t}} \left(Q \left(w _ {t}\right) - \hat {Q} _ {\theta_ {t} ^ {*}}\right) + 1 6 \alpha^ {w} (1 + \gamma) L _ {Q} H _ {Q} \sigma_ {w} ^ {3} + 4 \alpha^ {w} (1 + \gamma) L _ {Q} \epsilon \sigma_ {w}, \\ \end{array}
$$
where the second line is by the definition in Eq.(16), the sixth line is by Lemma (C.2) and Lemma C.5, and the eighth line is by Lemma C.1. Here,  $\lambda \in [0,1]$  is some scalar and  $w_{\mathrm{mid}} = \lambda w_t + (1 - \lambda)\hat{w}_{\theta_t}^*$ . So we arrive at the final bound for  $I_{1,1}$ :
$$
\begin{array}{l} 2 \alpha^ {w} (w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}) ^ {T} \bar {f} (w _ {t}, \theta_ {t}) \leq - 2 \alpha^ {w} \mathcal {N} _ {\theta_ {t}} (Q (w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*}) \\ + 1 6 \alpha^ {w} (1 + \gamma) L _ {Q} H _ {Q} \sigma_ {w} ^ {3} + 4 \alpha^ {w} (1 + \gamma) L _ {Q} \epsilon \sigma_ {w}. \\ \end{array}
$$
To analyze  $I_{1,2}$ , we conclude
$$
\mathbb {E} \left[ \alpha^ {w ^ {2}} | | f (O _ {t}, w _ {t}) | | ^ {2} \right] \leq \alpha^ {w ^ {2}} U _ {f} ^ {2}.
$$
To analyze  $I_{1,3}$ , for simplicity, we denote  $F(O_t, w, \theta) = (w - \hat{w}_\theta^*)^T [f(O_t, w) - \bar{f}(w, \theta)]$ . We have
$$
\begin{array}{l} F (O _ {t}, w _ {t}, \theta_ {t}) = \underbrace {F (O _ {t} , w _ {t} , \theta_ {t}) - F (O _ {t} , w _ {t} , \theta_ {t - \tau_ {\mathrm {m i x}}})} _ {J _ {1}} \\ + \underbrace {F (O _ {t} , w _ {t} , \theta_ {t - \tau_ {\mathrm {m i x}}}) - F (O _ {t} , w _ {t - \tau_ {\mathrm {m i x}}} , \theta_ {t - \tau_ {\mathrm {m i x}}})} _ {J _ {2}} \\ + \underbrace {F (O _ {t} , w _ {t - \tau_ {\mathrm {m i x}}} , \theta_ {t - \tau_ {\mathrm {m i x}}}) - F (\tilde {O} _ {t} , w _ {t - \tau_ {\mathrm {m i x}}} , \theta_ {t - \tau_ {\mathrm {m i x}}})} _ {J _ {3}} \\ + \underbrace {F (\tilde {O} _ {t} , w _ {t - \tau_ {\mathrm {m i x}}} , \theta_ {t - \tau_ {\mathrm {m i x}}})} _ {J _ {4}}. \\ \end{array}
$$
For  $J_{1}$ , we have
$$
\begin{array}{l} \left| F \left(O _ {t}, w _ {t}, \theta_ {t}\right) - F \left(O _ {t}, w _ {t}, \theta_ {t - \tau_ {\mathrm {m i x}}}\right) \right| \leq L _ {F _ {\theta}} \left\| \theta_ {t} - \theta_ {t - \tau_ {\mathrm {m i x}}} \right\| \\ \leq \frac {\alpha^ {\theta} \tau_ {\operatorname {m i x}} L _ {F _ {\theta}} U _ {g}}{1 - \gamma}. \\ \end{array}
$$
For  $J_{2}$ , we have
$$
\begin{array}{l} \left| F \left(O _ {t}, w _ {t}, \theta_ {t - \tau_ {\operatorname {m i x}}}\right) - F \left(O _ {t}, w _ {t - \tau_ {\operatorname {m i x}}}, \theta_ {t - \tau_ {\operatorname {m i x}}}\right) \right| \leq L _ {F _ {w}} | | w _ {t} - w _ {t - \tau_ {\operatorname {m i x}}} | | \\ \leq \alpha^ {w} \tau_ {\mathrm {m i x}} L _ {F _ {w}} \sigma_ {w}. \\ \end{array}
$$
For  $J_{3}$ , by Lemma C.17, we have
$$
\begin{array}{l} \left| \mathbb {E} \left[ F (O _ {t}, w _ {t - \tau_ {\mathrm {m i x}}}, \theta_ {t - \tau_ {\mathrm {m i x}}}) - F (\tilde {O} _ {t}, w _ {t - \tau_ {\mathrm {m i x}}}, \theta_ {t - \tau_ {\mathrm {m i x}}}) | \theta_ {t - \tau_ {\mathrm {m i x}}}, s _ {t - \tau_ {\mathrm {m i x}} + 1} \right] \right| \\ \leq 4 \sigma_ {w} U _ {f} n _ {a} L _ {\pi} \sum_ {i = t - \tau_ {\mathrm {m i x}}} ^ {t} \mathbb {E} \left[ \left\| \theta_ {i} - \theta_ {t - \tau_ {\mathrm {m i x}}} \right\| | \theta_ {t - \tau_ {\mathrm {m i x}}}, s _ {t - \tau_ {\mathrm {m i x}} + 1} \right] \\ \leq \frac {4 \alpha^ {\theta} \sigma_ {w} U _ {f} n _ {a} L _ {\pi} \tau_ {\mathrm {m i x}} (\tau_ {\mathrm {m i x}} - 1) U _ {g}}{1 - \gamma}, \\ \end{array}
$$
$$
\left| \mathbb {E} \left[ F (O _ {t}, w _ {t - \tau_ {\mathrm {m i x}}}, \theta_ {t - \tau_ {\mathrm {m i x}}}) - F (\tilde {O} _ {t}, w _ {t - \tau_ {\mathrm {m i x}}}, \theta_ {t - \tau_ {\mathrm {m i x}}}) \right] \right| \leq 4 \alpha^ {\theta} \sigma_ {w} U _ {f} n _ {a} L _ {\pi} \tau_ {\mathrm {m i x}} (\tau_ {\mathrm {m i x}} - 1) U _ {g}.
$$
For  $J_{4}$ , by Lemma C.17, we have
$$
\left| \mathbb {E} \left[ F (\tilde {O} _ {t}, w _ {t - \tau_ {\mathrm {m i x}}}, \theta_ {t - \tau_ {\mathrm {m i x}}}) | \theta_ {t - \tau_ {\mathrm {m i x}}}, s _ {t - \tau_ {\mathrm {m i x}} + 1} \right] \right| \leq 2 \sigma_ {w} U _ {f} C \beta^ {\tau_ {\mathrm {m i x}} - 1},
$$
Hence, for  $I_{1,3}$ , we have
$$
\mathbb {E} \left[ F (O _ {t}, w _ {t}, \theta_ {t}) \right] \leq \frac {\alpha^ {\theta} \tau_ {\mathrm {m i x}} \left(L _ {F _ {\theta}} + 4 \sigma_ {w} U _ {f} n _ {a} L _ {\pi} (\tau_ {\mathrm {m i x}} - 1)\right) U _ {g}}{1 - \gamma} + \alpha^ {w} \tau_ {\mathrm {m i x}} L _ {F _ {w}} \sigma_ {w} + 2 \sigma_ {w} U _ {f} C \beta^ {\tau_ {\mathrm {m i x}} - 1},
$$
$$
\begin{array}{l} \mathbb {E} \left[ 2 \alpha^ {w} (w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}) ^ {T} \left[ f (O _ {t}, w _ {t}) - \bar {f} (w _ {t}, \theta_ {t}) \right] \right] \leq \frac {2 \alpha^ {w} \alpha^ {\theta} \tau_ {\mathrm {m i x}} \left(L _ {F _ {\theta}} + 4 \sigma_ {w} U _ {f} n _ {a} L _ {\pi} (\tau_ {\mathrm {m i x}} - 1)\right) U _ {g}}{1 - \gamma} \\ + 2 \alpha^ {w 2} \tau_ {\mathrm {m i x}} L _ {F _ {w}} \sigma_ {w} + 4 \alpha^ {w} \sigma_ {w} U _ {f} C \beta^ {\tau_ {\mathrm {m i x}} - 1}. \\ \end{array}
$$
Hence, for  $I_{1}$
$$
\begin{array}{l} \mathbb {E} \left[ | | w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} f (O _ {t}, w _ {t}) | | ^ {2} \right] \leq - 2 \alpha^ {w} \mathbb {E} \left[ \mathcal {N} _ {\theta_ {t}} (Q (w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*}) \right] + \mathbb {E} \left[ | | w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} | | ^ {2} \right] \\ + 1 6 \alpha^ {w} (1 + \gamma) L _ {Q} H _ {Q} \sigma_ {w} ^ {3} + 4 \alpha^ {w} (1 + \gamma) L _ {Q} \epsilon \sigma_ {w} \\ + \alpha^ {w 2} U _ {f} ^ {2} + \frac {2 \alpha^ {w} \alpha^ {\theta} \tau_ {\mathrm {m i x}} \left(L _ {F _ {\theta}} + 4 \sigma_ {w} U _ {f} n _ {a} L _ {\pi} (\tau_ {\mathrm {m i x}} - 1)\right) U _ {g}}{1 - \gamma} \\ + 2 \alpha^ {w 2} \tau_ {\mathrm {m i x}} L _ {F _ {w}} \sigma_ {w} + 4 \alpha^ {w} \sigma_ {w} U _ {f} C \beta^ {\tau_ {\mathrm {m i x}} - 1} \\ = - 2 \alpha^ {w} \mathbb {E} \left[ \mathcal {N} _ {\theta_ {t}} \left(Q \left(w _ {t}\right) - \hat {Q} _ {\theta_ {t}} ^ {*}\right) \right] + \mathbb {E} \left[ \left\| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} \right\| ^ {2} \right] + C _ {w, 1} + C _ {w, 2}, \tag {19} \\ \end{array}
$$
$$
C _ {w, 1} = 1 6 \alpha^ {w} (1 + \gamma) L _ {Q} H _ {Q} \sigma_ {w} ^ {3} + 4 \alpha^ {w} (1 + \gamma) L _ {Q} \epsilon \sigma_ {w} + \alpha^ {w 2} U _ {f} ^ {2}
$$
$$
C _ {w, 2} = \frac {2 \alpha^ {w} \alpha^ {\theta} \tau_ {\mathrm {m i x}} \left(L _ {F _ {\theta}} + 4 \sigma_ {w} U _ {f} n _ {a} L _ {\pi} (\tau_ {\mathrm {m i x}} - 1)\right) U _ {g}}{1 - \gamma} + 2 \alpha^ {w 2} \tau_ {\mathrm {m i x}} L _ {F _ {w}} \sigma_ {w} + 4 \alpha^ {w} \sigma_ {w} U _ {f} C \beta^ {\tau_ {\mathrm {m i x}} - 1}.
$$
To analyze  $I_{2}$ , we derive
$$
\begin{array}{l} \mathbb {E} \left[ \left(\hat {w} _ {\theta_ {t}} ^ {*} - \hat {w} _ {\theta_ {t + 1}} ^ {*}\right) ^ {T} \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} f \left(O _ {t}, w _ {t}\right)\right) \right] \\ = \underbrace {\mathbb {E} \left[ (\hat {w} _ {\theta_ {t}} ^ {*} - \hat {w} _ {\theta_ {t + 1}} ^ {*}) ^ {T} (w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} (w _ {t} , \theta_ {t})) \right]} _ {I _ {2, 1}} + \underbrace {\mathbb {E} \left[ (\hat {w} _ {\theta_ {t}} ^ {*} - \hat {w} _ {\theta_ {t + 1}} ^ {*}) ^ {T} \alpha^ {w} \left[ f (O _ {t} , w _ {t}) - \bar {f} (w _ {t} , \theta_ {t}) \right] \right]} _ {I _ {2, 2}}. \\ \end{array}
$$
To analyze  $I_{2,1}$  first, by lemma C.18, we already know
$$
\begin{array}{l} \mathbb {E} \left[ \left(\hat {w} _ {\theta_ {t}} ^ {*} - \hat {w} _ {\theta_ {t + 1}} ^ {*}\right) ^ {T} (w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} (w _ {t}, \theta_ {t})) | \mathcal {F} _ {t} \right] \\ \leq \underbrace {\left(\sqrt {2} \alpha^ {\theta} L _ {w} | | \bar {g} (w _ {t} , \theta_ {t}) | | + \frac {\alpha^ {\theta^ {2}} H _ {w} U _ {g} ^ {2}}{\sqrt {2} (1 - \gamma) ^ {2}}\right)} _ {I _ {2, 1, 1}} \cdot \underbrace {\left\| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} (w _ {t} , \theta_ {t}) \right\|}. \\ \end{array}
$$
For  $I_{2,1,2}$ , a rough bound would be
$$
\left\| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} (w _ {t}, \theta_ {t}) \right\| ^ {2} = \| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} \| ^ {2} + 2 \alpha^ {w} (w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}) ^ {T} \bar {f} (w _ {t}, \theta_ {t}) + \alpha^ {w ^ {2}} | | \bar {f} (w _ {t}, \theta_ {t}) | | ^ {2}.
$$
Notice that this term is very similar to what we have in Eq.(18). The two difference are (a) all the  $f$  in Eq.(18) are replaced by  $\bar{f}$  and (b) expectation is removed. In this case,  $I_{1,3}$  will be 0 and a similar bound for  $I_{1,1}$  and  $I_{1,2}$  will also hold. This implies the following result:
$$
\begin{array}{l} \left\| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} \bar {f} (w _ {t}, \theta_ {t}) \right\| ^ {2} \leq - 2 \alpha^ {w} \mathcal {N} _ {\theta_ {t}} (Q (w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*}) + \| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} \| ^ {2} + 1 6 \alpha^ {w} (1 + \gamma) L _ {Q} H _ {Q} \sigma_ {w} ^ {3} \\ + 4 \alpha^ {w} (1 + \gamma) L _ {Q} \epsilon \sigma_ {w} + \alpha^ {w 2} U _ {f} ^ {2} \\ \leq \left(\frac {1}{\lambda_ {\min }} - 2 \alpha^ {w}\right) \mathcal {N} _ {\theta_ {t}} \left(Q \left(w _ {t}\right) - \hat {Q} _ {\theta_ {t}} ^ {*}\right) + C _ {w, 1}, \tag {20} \\ \end{array}
$$
where the last line uses Lemma C.3.
For  $I_{2,1,1}$ , recall that
$$
\begin{array}{l} \left\| \bar {g} \left(w _ {t}, \theta_ {t}\right) \right\| \leq \left\| \nabla V _ {\theta_ {t}} ^ {*} \right\| + \left\| D _ {Q} \right\| \\ \leq \left\| \nabla V _ {\theta_ {t}} ^ {*} \right\| + \sqrt {c _ {1} \mathcal {N} _ {\theta_ {t}} (Q (w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*}) + c _ {2} \epsilon^ {2}}. \\ \end{array}
$$
Hence, we have the following bound for  $I_{2,1,1}$ :
$$
\begin{array}{l} \sqrt {2} \alpha^ {\theta} L _ {w} | | \bar {g} (w _ {t}, \theta_ {t}) | | + \frac {\alpha^ {\theta^ {2}} H _ {w} U _ {g} ^ {2}}{\sqrt {2} (1 - \gamma) ^ {2}} \\ \leq \sqrt {2} \alpha^ {\theta} L _ {w} \left\| \nabla V _ {\theta_ {t}} ^ {*} \right\| + \sqrt {2} \alpha^ {\theta} L _ {w} \sqrt {c _ {1} \mathcal {N} _ {\theta_ {t}} (Q (w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*}) + c _ {2} \epsilon^ {2}} + \frac {\alpha^ {\theta^ {2}} H _ {w} U _ {g} ^ {2}}{\sqrt {2} (1 - \gamma) ^ {2}}. \\ \end{array}
$$
Hence, we have the following result: