SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Lemma D.12. For  $i \in \{1, 2\}$ , we have from Eq. (42), Lemma D.10, and Lemma D.11 that
$$
\begin{array}{l} \mathbb {E} [ \| q _ {k + 1} ^ {i} - \bar {q} _ {k + 1} ^ {i} \| _ {2} ^ {2} ] \leq (1 - 2 \alpha_ {k} c _ {\tau}) \mathbb {E} [ \| q _ {k} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {2} ^ {2} ] + \frac {4 | \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {2}} (\alpha_ {k} ^ {2} + 2 \alpha_ {k} \beta_ {k} + \beta_ {k} ^ {2}) \\ + \frac {3 4 | \mathcal {S} | A _ {\max } ^ {2} \beta_ {k}}{\tau (1 - \gamma) ^ {2}} \mathbb {E} [ \| q _ {k} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {2} ^ {2} ] + \frac {\beta_ {k}}{8} \mathbb {E} [ \mathcal {L} _ {\pi} (k) ] + \frac {3 4 z _ {k} \alpha_ {k} \alpha_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}} \\ \leq \left(1 - 2 \alpha_ {k} c _ {\tau} + \frac {3 4 | \mathcal {S} | A _ {\max } ^ {2} \beta_ {k}}{\tau (1 - \gamma) ^ {2}}\right) \mathbb {E} [ \| q _ {k} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {2} ^ {2} ] + \frac {\beta_ {k}}{8} \mathbb {E} [ \mathcal {L} _ {\pi} (k) ] \\ + \frac {5 0 | \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {2}} z _ {k} \alpha_ {k} \alpha_ {k - z _ {k}, k - 1}, \\ \end{array}
$$
where the second inequality follows from  $\beta_{k} = c_{\alpha ,\beta}\alpha_{k}$  with  $c_{\alpha ,\beta}\leq 1$  . Since
$$
c _ {\alpha , \beta} \leq \frac {c _ {\tau} \tau (1 - \gamma) ^ {2}}{3 4 | \mathcal {S} | A _ {\max } ^ {2}}, \tag {Condition 3.1}
$$
$$
\mathbb {E} [ \| q _ {k + 1} ^ {i} - \bar {q} _ {k + 1} ^ {i} \| _ {2} ^ {2} ] \leq (1 - \alpha_ {k} c _ {\tau}) \mathbb {E} [ \| q _ {k} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {2} ^ {2} ] + \frac {\beta_ {k}}{8} \mathbb {E} [ \mathcal {L} _ {\pi} (k) ] + \frac {5 0 | \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {2}} z _ {k} \alpha_ {k} \alpha_ {k - z _ {k}, k - 1}.
$$
Summing up the previous inequality for  $i = 1,2$ , we have
$$
\mathbb {E} [ \mathcal {L} _ {q} (k + 1) ] \leq (1 - \alpha_ {k} c _ {\tau}) \mathbb {E} [ \mathcal {L} _ {q} (k) ] + \frac {\beta_ {k}}{4} \mathbb {E} [ \mathcal {L} _ {\pi} (k) ] + \frac {1 0 0 | \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {2}} z _ {k} \alpha_ {k} \alpha_ {k - z _ {k}, k - 1}.
$$
D.6 Solving Coupled Lyapunov Drift Inequalities
We first restate the Lyapunov drift inequalities from previous sections. Recall our notation  $\mathcal{L}_q(t,k) = \sum_{i=1,2} \| q_{t,k}^i - \vec{q}_{t,k}^i \|_2^2$ ,  $\mathcal{L}_{\pi}(t,k) = \max_{s \in S} V_{v_t,s}(\pi_{t,k}^1(s), \pi_{t,k}^2(s))$ ,  $\mathcal{L}_{\mathrm{sum}}(t) = \| v_t^1 + v_t^2 \|_\infty$ , and  $\mathcal{L}_v(t) = \sum_{i=1,2} \| v_t^i - v_*^i \|_\infty$ . Let  $\mathcal{F}_t$  be the history of Algorithm 2 right before the  $t$ -th outer-loop iteration. Note that  $v_t^1$  and  $v_t^2$  are both measurable with respect to  $\mathcal{F}_t$ . In what follows, for ease of presentation, we write  $\mathbb{E}_t[\cdot]$  for  $\mathbb{E}[\cdot | \mathcal{F}_t]$ .
- Lemma D.5: It holds for all  $t \geq 0$  that
$$
\mathcal {L} _ {v} (t + 1) \leq \gamma \mathcal {L} _ {v} (t) + 4 \mathcal {L} _ {\text {s u m}} (t) + 2 \mathcal {L} _ {q} ^ {1 / 2} (t, K) + 4 \mathcal {L} _ {\pi} (t, K) + 6 \tau \log \left(A _ {\max }\right). \tag {44}
$$
- Lemma D.6: It holds for all  $t \geq 0$  that
$$
\mathcal {L} _ {\text {s u m}} (t + 1) \leq \gamma \mathcal {L} _ {\text {s u m}} (t) + 2 \mathcal {L} _ {q} ^ {1 / 2} (t, K). \tag {45}
$$
- Lemma D.8: It holds for all  $t, k \geq 0$  that
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k + 1) \right] \leq \left(1 - \frac {3 \beta_ {k}}{4}\right) \mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k) \right] + \frac {1 6 A _ {\max } ^ {2} \beta_ {k}}{\tau} \mathcal {L} _ {\operatorname {s u m}} (t) ^ {2} \\ + \frac {3 2 A _ {\operatorname* {m a x}} ^ {2} \beta_ {k}}{\tau^ {3} \ell_ {\tau} ^ {2} (1 - \gamma) ^ {2}} \mathcal {L} _ {q} (k) + 2 L _ {\tau} \beta_ {k} ^ {2}. \tag {46} \\ \end{array}
$$
- Lemma D.12: It holds for all  $t \geq 0$  and  $k \geq z_k$  that
$$
\mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, k + 1) \right] \leq \left(1 - \alpha_ {k} c _ {\tau}\right) \mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, k) \right] + \frac {\beta_ {k}}{4} \mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k) \right] + \frac {1 0 0 | S | A _ {\max }}{(1 - \gamma) ^ {2}} z _ {k} \alpha_ {k} \alpha_ {k - z _ {k}, k - 1}. \tag {47}
$$
Adding up Eqs. (46) and (47), we have by  $c_{\alpha,\beta} \leq \min \left(\frac{1}{L_\tau^{1/2}}, \frac{c_\tau \tau^3 \ell_\tau^2 (1 - \gamma)^2}{128 A_{\max}^2}, c_\tau\right)$  (cf. Condition 3.1) that
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k + 1) + \mathcal {L} _ {q} (t, k + 1) \right] \leq \left(1 - \frac {\beta_ {k}}{2}\right) \mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k) + \mathcal {L} _ {q} (t, k) \right] \\ \frac {1 6 A _ {\operatorname* {m a x}} ^ {2} \beta_ {k}}{\tau} \mathcal {L} _ {\text {s u m}} (t) ^ {2} + \frac {1 0 2 | S | A _ {\max }}{(1 - \gamma) ^ {2}} z _ {k} \alpha_ {k} \alpha_ {k - z _ {k}, k - 1}. \tag {48} \\ \end{array}
$$
D.6.1 Constant Stepsize
When using constant stepsizes, i.e.,  $\alpha_{k}\equiv \alpha$ ,  $\beta_{k}\equiv \beta = c_{\alpha ,\beta}\alpha$ , repeatedly using Eq. (48) from  $z_{\beta}$  to  $k$ , we have
$$
\mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k) + \mathcal {L} _ {q} (t, k) \right] \leq \left(1 - \frac {\beta}{2}\right) ^ {k - z _ {\beta}} \left(\mathcal {L} _ {\pi} (t, 0) + \mathcal {L} _ {q} (t, 0)\right)
$$
$$
+ \frac {1 6 A _ {\max } ^ {2}}{\tau} \mathcal {L} _ {\operatorname {s u m}} (t) ^ {2} + \frac {2 0 4 | \mathcal {S} | A _ {\max }}{(1 - \gamma) ^ {2} c _ {\alpha , \beta}} z _ {\beta} ^ {2} \alpha . \tag {49}
$$
We next bound  $\mathcal{L}_{\pi}(t,0) + \mathcal{L}_q(t,0)$ . For  $i\in \{1,2\}$ , we have
$$
\begin{array}{l} \mathcal {L} _ {\pi} (t, 0) = \max  _ {s} V _ {v _ {t}, s} \left(\pi_ {t, 0} ^ {1} (s), \pi_ {t, 0} ^ {2} (s)\right) \\ = \max _ {s} \sum_ {i = 1, 2} \max _ {\mu^ {i}} \{(\mu^ {i} - \pi_ {t, 0} ^ {i} (s)) ^ {\top} \mathcal {T} ^ {i} (v _ {t} ^ {i}) (s) \pi_ {t, 0} ^ {- i} (s) + \tau \nu (\mu^ {i}) - \tau \nu (\pi_ {t, 0} ^ {i} (s)) \} \\ \leq 2 \sum_ {i = 1, 2} \max _ {s, a ^ {i}, a ^ {- i}} | \mathcal {T} ^ {i} (v _ {t} ^ {i}) (s, a ^ {i}, a ^ {- i}) | + 2 \tau \log (A _ {\max }) \\ \leq \frac {4}{(1 - \gamma)} + 2 \tau \log (A _ {\max}), \\ \end{array}
$$
$$
\mathcal {L} _ {q} (t, 0) = \sum_ {i = 1, 2} \| q _ {t, 0} ^ {i} - \bar {q} _ {t, 0} ^ {i} \| _ {2} ^ {2} \leq \frac {8 | \mathcal {S} | A _ {\max }}{(1 - \gamma) ^ {2}}. \tag {Lemma D.1}
$$
$$
\mathcal {L} _ {\pi} (t, 0) + \mathcal {L} _ {q} (t, 0) \leq \frac {4}{(1 - \gamma)} + 2 \tau \log (A _ {\max }) + \frac {8 | \mathcal {S} | A _ {\max }}{(1 - \gamma) ^ {2}} = L _ {\mathrm {i n}}.
$$
Using the previous inequality in Eq. (49), we have
$$
\mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k) + \mathcal {L} _ {q} (t, k) \right] \leq L _ {\text {i n}} \left(1 - \frac {\beta}{2}\right) ^ {k - z _ {\beta}} + \frac {1 6 A _ {\max } ^ {2}}{\tau} \mathcal {L} _ {\text {s u m}} (t) ^ {2} + \frac {2 0 4 | S | A _ {\max }}{(1 - \gamma) ^ {2} c _ {\alpha , \beta}} z _ {\beta} ^ {2} \alpha , \tag {50}
$$
$$
\mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k) \right] \leq L _ {\text {i n}} \left(1 - \frac {\beta}{2}\right) ^ {k - z _ {\beta}} + \frac {1 6 A _ {\max } ^ {2}}{\tau} \mathcal {L} _ {\text {s u m}} (t) ^ {2} + \frac {2 0 4 | S | A _ {\max }}{(1 - \gamma) ^ {2} c _ {\alpha , \beta}} z _ {\beta} ^ {2} \alpha .
$$
Substituting the previous inequality on  $\mathbb{E}_t[\mathcal{L}_{\pi}(t,k)]$  into Eq. (47), we have
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, k + 1) \right] \leq (1 - \alpha c _ {\tau}) \mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, k) \right] + \frac {1 5 1 | \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {2}} z _ {\beta} ^ {2} \alpha^ {2} + \frac {\beta L _ {\text {i n}}}{4} \left(1 - \frac {\beta}{2}\right) ^ {k - z _ {\beta}} \\ + \frac {4 A _ {\mathrm {m a x}} ^ {2} \beta}{\tau} \mathcal {L} _ {\mathrm {s u m}} (t) ^ {2}. \\ \end{array}
$$
Repeatedly using the previous inequality, since  $c_{\alpha,\beta} \leq c_{\tau}$  (cf. Condition 3.1), we have
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, k) \right] \leq L _ {\text {i n}} \left(1 - c _ {\tau} \alpha\right) ^ {k - z _ {\beta}} + \frac {\beta L _ {\text {i n}} (k - z _ {\beta})}{4} \left(1 - \frac {\beta}{2}\right) ^ {k - z _ {\beta} - 1} \\ + \frac {4 A _ {\operatorname* {m a x}} ^ {2} c _ {\alpha , \beta}}{c _ {\tau} \tau} \mathcal {L} _ {\operatorname {s u m}} (t) ^ {2} + \frac {1 5 1 | \mathcal {S} | A _ {\operatorname* {m a x}}}{(1 - \gamma) ^ {2} c _ {\tau}} z _ {\beta} ^ {2} \alpha , \\ \end{array}
$$
which implies (by using Jensen's inequality) that
$$
\begin{array}{l} \mathbb {E} _ {t} [ \mathcal {L} _ {q} (t, k) ^ {1 / 2} ] \leq L _ {\mathrm {i n}} ^ {1 / 2} \left(1 - c _ {\tau} \alpha\right) ^ {\frac {k - z _ {\beta}}{2}} + \frac {\beta^ {1 / 2} L _ {\mathrm {i n}} ^ {1 / 2} (k - z _ {\beta}) ^ {1 / 2}}{2} \left(1 - \frac {\beta}{2}\right) ^ {\frac {k - z _ {\beta} - 1}{2}} \\ + \frac {2 A _ {\operatorname* {m a x}} c _ {\alpha , \beta} ^ {1 / 2}}{c _ {\tau} ^ {1 / 2} \tau^ {1 / 2}} \mathcal {L} _ {\operatorname {s u m}} (t) + \frac {1 3 | \mathcal {S} | ^ {1 / 2} A _ {\operatorname* {m a x}} ^ {1 / 2}}{(1 - \gamma) c _ {\tau} ^ {1 / 2}} z _ {\beta} \alpha^ {1 / 2}. \\ \end{array}
$$
Substituting the previous bound on  $\mathbb{E}_t[\mathcal{L}_q(t,k)^{1 / 2}]$  into Eq. (45) and then taking total expectation, we have
$$
\mathbb {E} \left[ \mathcal {L} _ {\text {s u m}} (t + 1) \right] \leq \gamma \mathcal {L} _ {\text {s u m}} (t) + 2 L _ {\text {i n}} ^ {1 / 2} \left(1 - c _ {\tau} \alpha\right) ^ {\frac {K - z _ {\beta}}{2}} + \beta^ {1 / 2} L _ {\text {i n}} ^ {1 / 2} \left(K - z _ {\beta}\right) ^ {1 / 2} \left(1 - \frac {\beta}{2}\right) ^ {\frac {K - z _ {\beta} - 1}{2}}
$$
$$
\begin{array}{l} + \frac {4 A _ {\operatorname* {m a x}} c _ {\alpha , \beta} ^ {1 / 2}}{c _ {\tau} ^ {1 / 2} \tau^ {1 / 2}} \mathcal {L} _ {\text {s u m}} (t) + \frac {2 6 | \mathcal {S} | ^ {1 / 2} A _ {\operatorname* {m a x}} ^ {1 / 2}}{(1 - \gamma) c _ {\tau} ^ {1 / 2}} z _ {\beta} \alpha^ {1 / 2} \\ \leq \left(\frac {1 + \gamma}{2}\right) \mathcal {L} _ {\text {s u m}} (t) + 2 L _ {\text {i n}} ^ {1 / 2} \left(1 - c _ {\tau} \alpha\right) ^ {\frac {K - z _ {\beta}}{2}} \\ + \beta^ {1 / 2} L _ {\mathrm {i n}} ^ {1 / 2} (K - z _ {\beta}) ^ {1 / 2} \left(1 - \frac {\beta}{2}\right) ^ {\frac {K - z _ {\beta} - 1}{2}} + \frac {2 6 | \mathcal {S} | ^ {1 / 2} A _ {\max } ^ {1 / 2}}{(1 - \gamma) c _ {\tau} ^ {1 / 2}} z _ {\beta} \alpha^ {1 / 2}, \\ \end{array}
$$
where the last line follows from  $c_{\alpha, \beta} \leq \frac{c_{\tau} \tau(1 - \gamma)^2}{64A_{\max}^2}$  (cf. Condition 3.1). Since  $\| v_0^1 + v_0^2 \|_{\infty} \leq \frac{2}{1 - \gamma}$ , repeatedly using the previous inequality, we have for all  $k \geq 0$  that
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {L} _ {\text {s u m}} (t) \right] \leq \frac {2}{1 - \gamma} \left(\frac {1 + \gamma}{2}\right) ^ {t} + \frac {4 L _ {\text {i n}} ^ {1 / 2} \left(1 - c _ {\tau} \alpha\right) ^ {\frac {K - z _ {\beta}}{2}}}{1 - \gamma} \\ + \frac {2 \beta^ {1 / 2} L _ {\mathrm {i n}} ^ {1 / 2} (K - z _ {\beta}) ^ {1 / 2}}{1 - \gamma} \left(1 - \frac {\beta}{2}\right) ^ {\frac {K - z _ {\beta} - 1}{2}} + \frac {5 2 | \mathcal {S} | ^ {1 / 2} A _ {\max } ^ {1 / 2}}{(1 - \gamma) ^ {2} c _ {\tau} ^ {1 / 2}} z _ {\beta} \alpha^ {1 / 2} \\ \leq \frac {2}{1 - \gamma} \left(\frac {1 + \gamma}{2}\right) ^ {t} + \frac {6 L _ {\text {i n}} ^ {1 / 2} (K - z _ {\beta}) ^ {1 / 2}}{1 - \gamma} \left(1 - \frac {\beta}{2}\right) ^ {\frac {K - z _ {\beta} - 1}{2}} \\ + \frac {5 2 | \mathcal {S} | ^ {1 / 2} A _ {\max } ^ {1 / 2}}{(1 - \gamma) ^ {2} c _ {\tau} ^ {1 / 2}} z _ {\beta} \alpha^ {1 / 2}. \tag {51} \\ \end{array}
$$
Now we have obtained finite-sample bounds for  $\mathcal{L}_q(t,k),\mathcal{L}_{\pi}(t,k)$  , and  $\mathcal{L}_{\mathrm{sum}}(t)$  . The next step is to use them in Eq. (44) to obtain finite-sample bounds for  $\mathcal{L}_v(t)$  . Specifically, using Eq. (44), Eq. (50), and Eq. (51), we have
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {L} _ {v} (t + 1) \right] \leq \gamma \mathbb {E} \left[ \mathcal {L} _ {v} (t) \right] + 4 \mathbb {E} \left[ \mathcal {L} _ {\operatorname {s u m}} (t) \right] + 2 \mathbb {E} \left[ \mathcal {L} _ {q} ^ {1 / 2} (t, K) \right] + 4 \mathbb {E} \left[ \mathcal {L} _ {\pi} (t, K) \right] + 6 \tau \log \left(A _ {\max }\right) \\ \leq \gamma \mathbb {E} [ \mathcal {L} _ {v} (t) ] + 2 L _ {\text {i n}} ^ {1 / 2} \left(1 - c _ {\tau} \alpha\right) ^ {\frac {K - z _ {\beta}}{2}} \\ + \beta^ {1 / 2} L _ {\mathrm {i n}} ^ {1 / 2} (K - z _ {\beta}) ^ {1 / 2} \left(1 - \frac {\beta}{2}\right) ^ {\frac {K - z _ {\beta} - 1}{2}} + \frac {2 6 | \mathcal {S} | ^ {1 / 2} A _ {\max } ^ {1 / 2}}{(1 - \gamma) c _ {\tau} ^ {1 / 2}} z _ {\beta} \alpha^ {1 / 2} \\ + 4 L _ {\text {i n}} \left(1 - \frac {\beta}{2}\right) ^ {K - z _ {\beta}} + \frac {8 1 6 | \mathcal {S} | A _ {\max }}{(1 - \gamma) ^ {2} c _ {\alpha , \beta}} z _ {\beta} ^ {2} \alpha + 6 \tau \log \left(A _ {\max }\right) \\ + \frac {2 6 6 A _ {\max } ^ {2}}{\tau (1 - \gamma) ^ {2}} \left(\frac {1 + \gamma}{2}\right) ^ {t} + \frac {7 9 8 A _ {\max } ^ {2} L _ {\text {i n}} ^ {1 / 2} (K - z _ {\beta}) ^ {1 / 2}}{(1 - \gamma) ^ {2} \tau} \left(1 - \frac {\beta}{2}\right) ^ {\frac {K - z _ {\beta} - 1}{2}} \\ + \frac {6 9 1 6 | \mathcal {S} | ^ {1 / 2} A _ {\max } ^ {5 / 2}}{(1 - \gamma) ^ {3} \tau c _ {\tau} ^ {1 / 2}} z _ {\beta} \alpha^ {1 / 2} \\ \leq \gamma \mathbb {E} [ \mathcal {L} _ {v} (t) ] + \frac {1 2 2 3 | \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {2} c _ {\alpha , \beta}} z _ {\beta} ^ {2} \alpha^ {1 / 2} + 6 \tau \log (A _ {\max }) \\ + \frac {2 6 6 A _ {\max} ^ {2}}{\tau (1 - \gamma) ^ {2}} \left(\frac {1 + \gamma}{2}\right) ^ {t} + \frac {8 0 5 A _ {\max} ^ {2} L _ {\mathrm {i n}} (K - z _ {\beta}) ^ {1 / 2}}{(1 - \gamma) ^ {2} \tau} \left(1 - \frac {\beta}{2}\right) ^ {\frac {K - z _ {\beta} - 1}{2}}. \\ \end{array}
$$
Repeatedly using the previous inequality from 0 to  $T - 1$  and then using  $\mathcal{L}_v(0) \leq \frac{4}{1 - \gamma}$ , we have
$$
\begin{array}{l} \mathcal {L} _ {v} (T) \leq \frac {2 7 0 A _ {\operatorname* {m a x}} ^ {2} T}{\tau (1 - \gamma) ^ {2}} \left(\frac {1 + \gamma}{2}\right) ^ {T - 1} \\ + \frac {8 0 5 A _ {\max } ^ {2} L _ {\operatorname {i n}} (K - z _ {\beta}) ^ {1 / 2}}{\tau (1 - \gamma) ^ {3}} \left(1 - \frac {\beta}{2}\right) ^ {\frac {K - z _ {\beta} - 1}{2}} \\ + \frac {1 2 2 3 | \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {3} c _ {\alpha , \beta}} z _ {\beta} ^ {2} \alpha^ {1 / 2} + \frac {6 \tau \log (A _ {\max})}{1 - \gamma}. \\ \end{array}
$$
Our next step is to use the bounds we obtained for  $\mathcal{L}_q(t,k),\mathcal{L}_{\pi}(t,k),\mathcal{L}_v(t)$  , and  $\mathcal{L}_{\mathrm{sum}}(t)$  in Lemma D.4. For simplicity of presentation, we use  $a\lesssim b$  to mean that there exists a numerical constant  $c$  such that  $a\le cb$  . Using the previous inequality, Eq. (50), and Eq. (51), we have
$$
\begin{array}{l} \mathbb {E} [ \mathrm {N G} (\pi_ {T, K} ^ {1}, \pi_ {T, K} ^ {2}) ] \leq \frac {8}{1 - \gamma} \mathcal {L} _ {\text {s u m}} (T) + \frac {4}{1 - \gamma} \mathcal {L} _ {v} (T) + \frac {4}{1 - \gamma} \mathcal {L} _ {\pi} (T, K) + \frac {8 \tau \log (A _ {\max })}{1 - \gamma} \\ \lesssim \frac {A _ {\operatorname* {m a x}} ^ {2} T}{\tau (1 - \gamma) ^ {3}} \left(\frac {1 + \gamma}{2}\right) ^ {T - 1} + \frac {A _ {\operatorname* {m a x}} ^ {2} L _ {\mathrm {i n}} (K - z _ {\beta}) ^ {1 / 2}}{\tau (1 - \gamma) ^ {4}} \left(1 - \frac {\beta}{2}\right) ^ {\frac {K - z _ {\beta} - 1}{2}} \\ + \frac {| \mathcal {S} | A _ {\mathrm {m a x}}}{(1 - \gamma) ^ {4} c _ {\alpha , \beta}} z _ {\beta} ^ {2} \alpha^ {1 / 2} + \frac {\tau \log (A _ {\mathrm {m a x}})}{(1 - \gamma) ^ {2}}. \\ \end{array}
$$
The proof of Theorem 3.1 (1) is complete.
D.6.2 Diminishing Stepsizes
Consider using linearly diminishing step sizes, i.e.,  $\alpha_{k} = \frac{\alpha}{k + h}$ ,  $\beta_{k} = \frac{\beta}{k + h}$ , and  $\beta = c_{\alpha,\beta}\alpha$ . Repeatedly using Eq. (48), we have for all  $k \geq k_0 := \min \{k' \mid k' \geq z_{k'}\}$  that
$$
\begin{array}{l} \mathbb{E}_{t}[\mathcal{L}_{\pi}(t,k) + \mathcal{L}_{q}(t,k)]\leq L_{\text{in}}\underbrace{\prod_{m = k_{0}}^{k - 1}\left(1 - \frac{\beta_{m}}{2}\right)}_{\hat{\mathcal{E}}_{1}} + \frac{204|\mathcal{S}|A_{\max}}{(1 - \gamma)^{2}}\underbrace{\sum_{n = k_{0}}^{k - 1}z_{n}^{2}\alpha_{n}^{2}\prod_{m = n + 1}^{k - 1}\left(1 - \frac{\beta_{m}}{2}\right)}_{\hat{\mathcal{E}}_{2}} \\ + \frac {1 6 A _ {\operatorname* {m a x}} ^ {2}}{\tau} \mathcal {L} _ {\operatorname {s u m}} (t) ^ {2} \underbrace {\sum_ {n = k _ {0}} ^ {k - 1} \beta_ {n} \prod_ {m = n + 1} ^ {k - 1} \left(1 - \frac {\beta_ {m}}{2}\right)} _ {\hat {\mathcal {E}} _ {3}}. \\ \end{array}
$$
Next, we evaluate the terms  $\{\hat{\mathcal{E}}_j\}_{1\leq j\leq 3}$ . Terms like  $\{\hat{\mathcal{E}}_j\}_{1\leq j\leq 3}$  have been well studied in the existing literature [24, 44, 65]. Specifically, we have from [65, Appendix A.2.] and  $\beta = 4$  that
$$
\hat {\mathcal {E}} _ {1} \leq \frac {k _ {0} + h}{k + h}, \quad \hat {\mathcal {E}} _ {2} \leq \frac {6 4 e z _ {k} ^ {2}}{(k + h) c _ {\alpha , \beta} ^ {2}}, \text {a n d} \hat {\mathcal {E}} _ {3} \leq 2.
$$
$$
\mathbb {E} _ {t} [ \mathcal {L} _ {\pi} (t, k) + \mathcal {L} _ {q} (t, k) ] \le L _ {\mathrm {i n}} \frac {k _ {0} + h}{k + h} + \frac {3 2 6 4 e | \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {2} c _ {\alpha , \beta}} z _ {k} ^ {2} \alpha_ {k} + \frac {3 2 A _ {\max} ^ {2}}{\tau} \mathcal {L} _ {\mathrm {s u m}} (t) ^ {2},
$$
$$
\mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k) \right] \leq L _ {\mathrm {i n}} \frac {k _ {0} + h}{k + h} + \frac {3 2 6 4 e | \mathcal {S} | A _ {\max }}{(1 - \gamma) ^ {2} c _ {\alpha , \beta}} z _ {k} ^ {2} \alpha_ {k} + \frac {3 2 A _ {\max } ^ {2}}{\tau} \mathcal {L} _ {\operatorname {s u m}} (t) ^ {2}. \tag {52}
$$
Using the previous inequality on  $\mathbb{E}_t[\mathcal{L}_{\pi}(t,k)]$  in Eq. (47), we have
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, k + 1) \right] \leq \left(1 - \alpha_ {k} c _ {\tau}\right) \mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, k) \right] + \frac {1 0 0 | \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {2}} z _ {k} \alpha_ {k} \alpha_ {k - z _ {k}, k - 1} \\ + \frac {L _ {\mathrm {i n}} c _ {\alpha , \beta} \alpha_ {k} ^ {2}}{4 \alpha_ {k _ {0}}} + \frac {8 1 6 e | \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {2}} z _ {k} ^ {2} \alpha_ {k} ^ {2} + \frac {8 A _ {\max} ^ {2} \beta_ {k}}{\tau} \mathcal {L} _ {\mathrm {s u m}} (t) ^ {2} \\ \leq \left(1 - \alpha_ {k} c _ {\tau}\right) \mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, k) \right] + \frac {1 0 1 7 e L _ {\mathrm {i n}} | S | A _ {\max}}{(1 - \gamma) ^ {2} \alpha_ {k _ {0}}} z _ {k} ^ {2} \alpha_ {k} ^ {2} \\ + \frac {8 A _ {\operatorname* {m a x}} ^ {2} \beta_ {k}}{\tau} \mathcal {L} _ {\operatorname {s u m}} (t) ^ {2}. \\ \end{array}
$$
Repeatedly using the previous inequality starting from  $k_{0}$ , since  $\alpha c_{\tau} \geq 1$  (cf. Condition 3.1), we have
$$
\mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, k) \right] \leq L _ {\text {i n}} \frac {k _ {0} + h}{k + h} + \frac {4 0 6 8 e ^ {2} L _ {\text {i n}} | \mathcal {S} | A _ {\max }}{(1 - \gamma) ^ {2} c _ {\tau} \alpha_ {k _ {0}}} z _ {k} ^ {2} \alpha_ {k} + \frac {8 A _ {\max } ^ {2} c _ {\alpha , \beta}}{c _ {\tau} \tau} \mathcal {L} _ {\text {s u m}} (t) ^ {2},
$$
$$
\mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, k) ^ {1 / 2} \right] \leq L _ {\text {i n}} ^ {1 / 2} \left(\frac {k _ {0} + h}{k + h}\right) ^ {1 / 2} + \frac {6 4 e L _ {\text {i n}} ^ {1 / 2} | \mathcal {S} | ^ {1 / 2} A _ {\max } ^ {1 / 2}}{(1 - \gamma) c _ {\tau} ^ {1 / 2} \alpha_ {k _ {0}} ^ {1 / 2}} z _ {k} \alpha_ {k} ^ {1 / 2} + \frac {3 A _ {\max } c _ {\alpha , \beta} ^ {1 / 2}}{c _ {\tau} ^ {1 / 2} \tau^ {1 / 2}} \mathcal {L} _ {\text {s u m}} (t). \tag {53}
$$
Taking total expectation on both sides of the previous inequality and then using the result in Eq. (45), we have
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {L} _ {\text {s u m}} (t + 1) \right] \leq \gamma \mathbb {E} \left[ \mathcal {L} _ {\text {s u m}} (t) \right] + 2 L _ {\text {i n}} ^ {1 / 2} \left(\frac {k _ {0} + h}{K + h}\right) ^ {1 / 2} + \frac {1 2 8 e | \mathcal {S} | ^ {1 / 2} A _ {\max } ^ {1 / 2}}{(1 - \gamma) c _ {\tau} ^ {1 / 2} \alpha_ {k _ {0}} ^ {1 / 2}} z _ {K} \alpha_ {K} ^ {1 / 2} \\ + \frac {6 A _ {\operatorname* {m a x}} c _ {\alpha , \beta} ^ {1 / 2}}{c _ {\tau} ^ {1 / 2} \tau^ {1 / 2}} \mathcal {L} _ {\text {s u m}} (t) \\ \leq \left(\frac {\gamma + 1}{2}\right) \mathbb {E} [ \mathcal {L} _ {\mathrm {s u m}} (t) ] + \frac {1 3 0 e L _ {\mathrm {i n}} ^ {1 / 2} | \mathcal {S} | ^ {1 / 2} A _ {\mathrm {m a x}} ^ {1 / 2}}{(1 - \gamma) c _ {\tau} ^ {1 / 2} \alpha_ {k _ {0}} ^ {1 / 2}} z _ {K} \alpha_ {K} ^ {1 / 2}, \\ \end{array}
$$
where the last line follows from  $c_{\alpha, \beta} \leq \frac{c_{\tau} \tau(1 - \gamma)^2}{144A_{\max}^2}$  (cf. Condition 3.1). Repeatedly using the previous inequality starting from 0, we have
$$
\mathbb {E} \left[ \mathcal {L} _ {\text {s u m}} (t) \right] \leq \frac {2}{1 - \gamma} \left(\frac {1 + \gamma}{2}\right) ^ {t} + \frac {2 6 0 e L _ {\mathrm {i n}} ^ {1 / 2} | \mathcal {S} | ^ {1 / 2} A _ {\max } ^ {1 / 2}}{\left(1 - \gamma\right) ^ {2} c _ {\tau} ^ {1 / 2} \alpha_ {k _ {0}} ^ {1 / 2}} z _ {K} \alpha_ {K} ^ {1 / 2}. \tag {54}
$$
The next step is to bound  $\mathcal{L}_v(t)$ . Recall from Eq. (44) that
$$
\mathbb {E} _ {t} \left[ \mathcal {L} _ {v} (t + 1) \right] \leq \gamma \mathcal {L} _ {v} (t) + 4 \mathcal {L} _ {\operatorname {s u m}} (t) + 2 \mathbb {E} _ {t} \left[ \mathcal {L} _ {q} ^ {1 / 2} (t, K) \right] + 4 \mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, K) \right] + 6 \tau \log \left(A _ {\max }\right).
$$
Using Eqs. (52), (53), and (54) in the previous inequality, we have
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {L} _ {v} (t + 1) \right] \leq \gamma \mathbb {E} \left[ \mathcal {L} _ {v} (t) \right] + 4 \mathbb {E} \left[ \mathcal {L} _ {\operatorname {s u m}} (t) \right] + 2 \mathbb {E} \left[ \mathcal {L} _ {q} ^ {1 / 2} (t, K) \right] + 4 \mathbb {E} \left[ \mathcal {L} _ {\pi} (t, K) \right] + 6 \tau \log \left(A _ {\max }\right) \\ \leq \gamma \mathbb {E} [ \mathcal {L} _ {v} (t) ] + \frac {1 3 0 e L _ {\mathrm {i n}} ^ {1 / 2} | \mathcal {S} | ^ {1 / 2} A _ {\mathrm {m a x}} ^ {1 / 2}}{(1 - \gamma) c _ {\tau} ^ {1 / 2} \alpha_ {k _ {0}} ^ {1 / 2}} z _ {K} \alpha_ {K} ^ {1 / 2} \\ + \frac {4 L _ {\mathrm {i n}} \alpha_ {K}}{\alpha_ {k _ {0}}} + \frac {1 3 0 5 6 e | \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {2} c _ {\alpha , \beta}} z _ {K} ^ {2} \alpha_ {K} + 6 \tau \log (A _ {\max}) \\ + \frac {5 2 2 A _ {\mathrm {m a x}} ^ {2}}{\tau (1 - \gamma) ^ {2}} \left(\frac {1 + \gamma}{2}\right) ^ {t} + \frac {6 7 8 6 0 e L _ {\mathrm {i n}} ^ {1 / 2} | \mathcal {S} | ^ {1 / 2} A _ {\mathrm {m a x}} ^ {5 / 2}}{(1 - \gamma) ^ {3} \tau c _ {\tau} ^ {1 / 2} \alpha_ {k _ {0}} ^ {1 / 2}} z _ {K} \alpha_ {K} ^ {1 / 2} \\ \leq \gamma \mathbb {E} [ \mathcal {L} _ {v} (t) ] + \frac {5 2 2 A _ {\max} ^ {2}}{\tau (1 - \gamma) ^ {2}} \left(\frac {1 + \gamma}{2}\right) ^ {t} + \frac {1 5 0 5 6 e L _ {\mathrm {i n}} | \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {2} \alpha_ {k _ {0}} ^ {1 / 2} c _ {\alpha , \beta}} z _ {K} ^ {2} \alpha_ {K} ^ {1 / 2} \\ + 6 \tau \log (A _ {\max}). \\ \end{array}
$$
Repeatedly using the previous inequality starting from 0 to  $T - 1$ , we have
$$
\mathcal {L} _ {v} (T) \leq \frac {5 2 6 A _ {\max} ^ {2} T}{\tau (1 - \gamma) ^ {2}} \left(\frac {1 + \gamma}{2}\right) ^ {T - 1} + \frac {1 5 0 5 6 e L _ {\mathrm {i n}} | \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {3} \alpha_ {k _ {0}} ^ {1 / 2} c _ {\alpha , \beta}} z _ {K} ^ {2} \alpha_ {K} ^ {1 / 2} + \frac {6 \tau \log (A _ {\max})}{1 - \gamma}
$$
Finally, using the previous inequality, Eq. (52), and Eq. (54) in Lemma D.4, we obtain