SlowGuess/NeurIPS_2023_OCR huggingface dataset:
for any  $q^i \in \mathbb{R}^{|\mathcal{S}||\mathcal{A}^i|}$  and  $i \in \{1,2\}$ , where  $F^i(\cdot)$  is some appropriately defined operator that captures the dynamics of the update equation; see Appendix D.5.2 for its definition. In the term (13), the random tuple  $(S_k,A_k^i,A_k^{-i},S_{k + 1})$  is the  $k$ -th sample from the time-inhomogeneous Markov chain  $\{(S_k,A_k^i,A_k^{-i},S_{k + 1})\}_{k\geq 0}$  generated by the time-varying joint policies  $\{\pi_k\}_{k\geq 0}$  and  $(\hat{S},\hat{A}^i,\hat{A}^{-i},\hat{S}^{\prime})$  is a random tuple such that  $S \sim \mu_k(\cdot)$ ,  $A^i \sim \pi_k^i (\cdot |S)$ ,  $A^{-i} \sim \pi_k^{-i}(\cdot |S)$ , and  $S^{\prime} \sim p(\cdot |S,A^{i},A^{-i})$ , where  $\mu_k(\cdot)$  denotes the unique stationary distribution of the Markov chain  $\{S_n\}_{n\geq 0}$  induced by the joint policy  $\pi_k$ . Lemma B.1 implies that  $\mu_k$  exists and is unique.
In the existing literature, when  $\{(S_k,A_k^i,A_k^{-i},S_{k + 1})\}$  is sampled either in an i.i.d. manner or forms an ergodic time-homogeneous Markov chain, there are techniques that successfully bound the term (13) [94, 24, 23]. To deal with time-inhomogeneous Markovian noise, building upon existing conditioning results [23, 24, 41, 76] and also Lemma B.1, we develop a refined conditioning argument to show that
$$
(1 3) = \mathcal {O} \left(z _ {k} \sum_ {n = k - z _ {k}} ^ {k - 1} \alpha_ {n}\right), \tag {See Lemma D.11}
$$
where  $z_{k} = t(\ell_{\tau}, \beta_{k})$  is a uniform upper bound on the  $\beta_{k}$  - the mixing time (i.e., the uniform mixing time with accuracy  $\beta_{k}$ , see Eq. (12)) of the Markov chain  $\{S_{n}\}_{n \geq 0}$  induced by an arbitrary joint policy from the algorithm trajectory. Suppose that we are using diminishing step sizes of  $\mathcal{O}(1 / k)$  decay rate (similar results hold for using constant step sizes). Then, the uniform mixing property from Lemma B.1 (2) implies that  $z_{k} = \mathcal{O}(\log(1 / k))$ . As a result, we have  $\lim_{k \to \infty} (13) \leq \lim_{k \to \infty} z_{k} \sum_{n=k-z_{k}}^{k-1} \alpha_{n} = 0$ , which provides us a way to control the term in (13). After successfully handling (13), we are able to establish a Lyapunov drift inequality of  $\mathcal{L}_{q}(t, k)$ :
$$
\mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, k + 1) \right] \leq \underbrace {(1 - \alpha_ {k} c _ {\tau}) \mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t , k) \right]} _ {\text {D r i f t}} + \underbrace {C _ {0} z _ {k} \alpha_ {k} \alpha_ {k - z _ {k} , k - 1} + \frac {\beta_ {k}}{4} \mathbb {E} \left[ \mathcal {L} _ {\pi} (t , k) \right]} _ {\text {A d d i t i v e E r r o r s}}, \tag {14}
$$
where  $C_0$  is a (problem-dependent) constant, and we use the notation  $\alpha_{k_1,k_2} \coloneqq \sum_{k = k_1}^{k_2}\alpha_k$  to simplify the notation. See Lemma D.12 for more details. When  $k$  is large, it can be shown that  $\alpha_{k - z_k,k - 1} \leq 2\alpha_kz_k$  [65, Appendix 1.8].
B.4 Solving Coupled Lyapunov Drift Inequalities
Until this point, we have established the Lyapunov drift inequalities for the individual  $v$ -functions, the sum of the  $v$ -functions, the policies, and the  $q$ -functions in Eqs. (7), (8), (10), and (14), respectively. The last challenge is to find a way of using these coupled inequalities to derive the finite-sample bound. To elaborate, we first restate all the Lyapunov drift inequalities in the following:
$$
\mathcal {L} _ {v} (t + 1) \leq \gamma \mathcal {L} _ {v} (t) + 4 \mathcal {L} _ {\text {s u m}} (t) + 4 \mathcal {L} _ {\pi} (t, K) + 2 \mathcal {L} _ {q} ^ {1 / 2} (t, K) + 6 \tau \log \left(A _ {\max }\right), \tag {15}
$$
$$
\mathcal {L} _ {\text {s u m}} (t + 1) \leq \gamma \mathcal {L} _ {\text {s u m}} (t) + 2 \mathcal {L} _ {q} ^ {1 / 2} (t, K), \tag {16}
$$
$$
\mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k + 1) \right] \leq \left(1 - 3 \beta_ {k} / 4\right) \mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k) \right] + C _ {1} \left(\beta_ {k} ^ {2} + \beta_ {k} \mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, k) \right] + \beta_ {k} \mathcal {L} _ {\operatorname {s u m}} ^ {2} (t)\right), \tag {17}
$$
$$
\mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, k + 1) \right] \leq \left(1 - c _ {\tau} \alpha_ {k}\right) \mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, k) \right] + \beta_ {k} \mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k) \right] / 4 + C _ {3} z _ {k} ^ {2} \alpha_ {k} ^ {2}. \tag {18}
$$
To decouple the Lyapunov inequalities stated above, our high-level ideas are (1) using the Lyapunov drift inequalities in a combined way instead of in a separate manner, and (2) a bootstrapping procedure
where we first derive a crude bound and then substitute the crude bound back into the Lyapunov drift inequalities to derive a tighter bound. We next present our approach.
For ease of presentation, for a scalar-valued quantity  $W$  that is a function of  $k$  and/or  $t$ , we say  $W = o_k(1)$  if  $\lim_{k\to \infty}W = 0$  and  $W = o_t(1)$  if  $\lim_{t\to \infty}W = 0$ . The explicit convergence rates of the  $o_k(1)$  term and the  $o_t(1)$  term will be revealed in the complete proof in Appendix D.6, but is not important for the illustration here.
Step 1. Adding up Eq. (17) and (18), using Condition 3.1, and then repeatedly using the resulting inequality, we obtain:
$$
\mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k) \right] \leq \mathbb {E} _ {t} \left[ \mathcal {L} _ {\pi} (t, k) + \mathcal {L} _ {q} (t, k) \right] = o _ {k} (1) + \mathcal {O} (1) \mathcal {L} _ {\text {s u m}} ^ {2} (t), \forall t, k. \tag {19}
$$
Step 2. Substituting the bound for  $\mathbb{E}_t[\mathcal{L}_{\pi}(t,k)]$  in Eq. (19) into Eq. (18) and repeatedly using the resulting inequality, and we obtain:
$$
\mathbb {E} _ {t} \left[ \mathcal {L} _ {q} (t, K) \right] = o _ {K} (1) + \mathcal {O} \left(c _ {\alpha , \beta}\right) \mathcal {L} _ {\text {s u m}} ^ {2} (t), \forall t,
$$
which in turn implies (by first using Jensen's inequality and then taking total expectation) that:
$$
\mathbb {E} \left[ \mathcal {L} _ {q} ^ {1 / 2} (t, K) \right] = o _ {K} (1) + \mathcal {O} \left(c _ {\alpha , \beta} ^ {1 / 2}\right) \mathbb {E} \left[ \mathcal {L} _ {\text {s u m}} (t) \right], \forall t, \tag {20}
$$
where we recall that  $c_{\alpha, \beta} = \beta_k / \alpha_k$  is the stepsize ratio. The fact that we are able to get a factor of  $\mathcal{O}(c_{\alpha, \beta}^{1/2})$  in front of  $\mathbb{E}[\mathcal{L}_{\mathrm{sum}}(t)]$  is crucial for the decoupling procedure.
Step 3. Taking total expectation on both sides of Eq. (16) and then using the upper bound of  $\mathbb{E}[\mathcal{L}_q^{1/2}(t, K)]$  we obtained in Eq. (20), we obtain
$$
\mathbb {E} \left[ \mathcal {L} _ {\text {s u m}} (t + 1) \right] \leq \left(\gamma + \mathcal {O} \left(c _ {\alpha , \beta} ^ {1 / 2}\right)\right) \mathbb {E} \left[ \mathcal {L} _ {\text {s u m}} (t) \right] + o _ {K} (1), \forall t.
$$
By choosing  $c_{\alpha, \beta}$  so that  $\mathcal{O}(c_{\alpha, \beta}^{1/2}) \leq (1 - \gamma)/2$ , the previous inequality implies
$$
\mathbb {E} \left[ \mathcal {L} _ {\text {s u m}} (t + 1) \right] \leq \left(1 - \frac {1 - \gamma}{2}\right) \mathbb {E} \left[ \mathcal {L} _ {\text {s u m}} (t) \right] + o _ {K} (1), \forall t, \tag {21}
$$
which can be repeatedly used to obtain
$$
\mathbb {E} \left[ \mathcal {L} _ {\text {s u m}} (t) \right] = o _ {t} (1) + o _ {K} (1). \tag {22}
$$
Substituting the previous bound on  $\mathbb{E}[\mathcal{L}_{\mathrm{sum}}(t)]$  into Eq. (19), we have
$$
\max  (\mathbb {E} [ \mathcal {L} _ {\pi} (t, K) ], \mathbb {E} [ \mathcal {L} _ {q} (t, K) ]) = o _ {t} (1) + o _ {K} (1). \tag {23}
$$
Step 4. Substituting the bounds we obtained for  $\mathbb{E}[\mathcal{L}_{\pi}(t,K)]$ ,  $\mathbb{E}[\mathcal{L}_q(t,K)]$ , and  $\mathbb{E}[\mathcal{L}_{\mathrm{sum}}(t)]$  in Eqs. (22) and (23) into Eq. (15), and then repeatedly using the resulting inequality from  $t = 0$  to  $t = T$ , we have
$$
\mathbb {E} \left[ \mathcal {L} _ {v} (T) \right] = o _ {T} (1) + o _ {K} (1) + \mathcal {O} (\tau).
$$
Now that we have obtained finite-sample bounds for  $\mathbb{E}[\mathcal{L}_v(T)],\mathbb{E}[\mathcal{L}_{\mathrm{sum}}(T)],\mathbb{E}[\mathcal{L}_\pi (T,K)]$  , and  $\mathbb{E}[\mathcal{L}_q(T,K)]$  , using them in Eq. (6), we finally obtain the desired finite-sample bound for the expected Nash gap.
Looking back at the decoupling procedure, Steps 2 and 3 are crucial. In fact, in Step 1 we already obtain a bound on  $\mathbb{E}_t[\mathcal{L}_q(t,k)]$ , where the additive error is  $\mathcal{O}(1)\mathbb{E}[\mathcal{L}_{\mathrm{sum}}(t)]$ . However, directly using this bound on  $\mathbb{E}_t[\mathcal{L}_q(t,k)]$  in Eq. (16) would result in an expansive inequality for  $\mathbb{E}[\mathcal{L}_{\mathrm{sum}}(t)]$ . By performing Step 2, we are able to obtain a tighter bound for  $\mathbb{E}_t[\mathcal{L}_q(t,k)]$ , with the additive error being  $\mathcal{O}(c_{\alpha,\beta}^{1/2})\mathbb{E}[\mathcal{L}_{\mathrm{sum}}(t)]$ . Furthermore, we can choose  $c_{\alpha,\beta}$  to be small enough so that after using the bound from Eq. (20) in Eq. (16), the additive error  $\mathcal{O}(c_{\alpha,\beta}^{1/2})\mathbb{E}[\mathcal{L}_{\mathrm{sum}}(t)]$  is dominated by the negative drift in Eq. (21).
C Proof of Theorem 2.1
The proof is divided into 4 steps. In Appendix C.1, we prove an important boundedness property regarding the iterates generated by Algorithm 1. In Appendices C.2 and C.3, we analyze the evolution of the policies and the  $q$ -functions by establishing the negative drift inequalities with respect to their associated Lyapunov functions. In Appendix C.4, we solve the coupled Lyapunov drift inequalities to prove Theorem 2.1. Moreover, we prove Corollary 2.1.1 in Appendix C.5. The statement and proof of all supporting lemmas used in this section are presented in Appendix C.6.
C.1 Boundedness of the Iterates
In this subsection, we show that the  $q$ -functions generated by Algorithm 1 are uniformly bounded from above, and the entries of the policies are uniformly bounded from below. The following lemma is needed to establish the result.
Lemma C.1. For any  $i\in \{1,2\}$  and  $q^i\in \mathbb{R}^{|\mathcal{A}|^i}$ , we have
$$
\min  _ {a ^ {i} \in \mathcal {A} ^ {i}} [ \sigma_ {\tau} (q ^ {i}) ] (a ^ {i}) \geq \frac {1}{(A _ {\max } - 1) \exp (2 \| q ^ {i} \| _ {\infty} / \tau) + 1}.
$$
Proof of Lemma C.1. Given  $i\in \{1,2\}$ , for any  $q^i\in \mathbb{R}^{|\mathcal{A}|^i}$  and  $a^i\in \mathcal{A}^i$ , we have
$$
\begin{array}{l} [ \sigma_ {\tau} (q ^ {i}) ] (a ^ {i}) = \frac {\exp (q ^ {i} (a ^ {i}) / \tau)}{\sum_ {\bar {a} ^ {i} \in \mathcal {A} ^ {i}} \exp (q ^ {i} (\bar {a} ^ {i}) / \tau)} \\ = \frac {1}{\sum_ {\bar {a} ^ {i} \neq a ^ {i}} \exp ((q ^ {i} (\bar {a} ^ {i}) - q ^ {i} (a ^ {i})) / \tau) + 1} \\ \geq \frac {1}{(| \mathcal {A} ^ {i} | - 1) \exp (2 \| q ^ {i} \| _ {\infty} / \tau) + 1} \\ \geq \frac {1}{(A _ {\max} - 1) \exp (2 \| q ^ {i} \| _ {\infty} / \tau) + 1}. \\ \end{array}
$$
Since the RHS of the previous inequality does not depend on  $a^i$ , we have the desired inequality.
We next derive the boundedness property in the following lemma.
Lemma C.2. It holds for all  $k \geq 0$  and  $i \in \{1,2\}$  that  $\| q_k^i \|_{\infty} \leq 1$  and  $\min_{a^i \in \mathcal{A}^i} \pi_k^i(a^i) \geq \ell_{\tau}$ , where  $\ell_{\tau} = [(A_{\max} - 1) \exp(2/\tau) + 1]^{-1}$ .
Proof of Lemma C.2. We prove the results by induction. Since  $q_0^i = 0$  and  $\pi_0^i$  is initialized as a uniform distribution on  $\mathcal{A}^i$ , we have the base case. Now suppose that the results hold for some  $k \geq 0$ . Using the update equation for  $q_k^i$  in Algorithm 1 Line 5, we have
$$
\begin{array}{l} | q _ {k + 1} ^ {i} (a ^ {i}) | = | (1 - \alpha_ {k} \mathbb {1} _ {\{a ^ {i} = A _ {k} ^ {i} \}}) q _ {k} ^ {i} (a ^ {i}) + \alpha_ {k} \mathbb {1} _ {\{a ^ {i} = A _ {k} ^ {i} \}} R _ {i} (A _ {k} ^ {i}, A _ {k} ^ {- i}) | \\ \leq \max  \left(\left| q _ {k} ^ {i} \left(a ^ {i}\right) \right|, \left(1 - \alpha_ {k}\right) \left| q _ {k} ^ {i} \left(a ^ {i}\right) \right| + \alpha_ {k} \mid R _ {i} \left(A _ {k} ^ {i}, A _ {k} ^ {- i}\right)\right) \\ \leq 1 \\ \end{array}
$$
for any  $a^i \in \mathcal{A}^i$ , where the last line follows from the induction hypothesis  $\|q_k^i\|_{\infty} \leq 1$  and  $|R_i(a^i, a^{-i})| \leq 1$  for all  $(a^i, a^{-i})$ . As for  $\pi_{k+1}^i$ , using the update equation for  $\pi_k^i$  in Algorithm 1, Line 3, we have
$$
\begin{array}{l} \pi_ {k + 1} ^ {i} \left(a ^ {i}\right) = \left(1 - \beta_ {k}\right) \pi_ {k} ^ {i} \left(a ^ {i}\right) + \beta_ {k} \left[ \sigma_ {\tau} \left(q _ {k} ^ {i}\right) \right] \left(a ^ {i}\right) \\ \geq \left(1 - \beta_ {k}\right) \ell_ {\tau} + \frac {\beta_ {k}}{\left(A _ {\max } - 1\right) \exp \left(2 \| q _ {k} ^ {i} \| _ {\infty} / \tau\right) + 1} \tag {Lemma C.1} \\ \geq \left(1 - \beta_ {k}\right) \ell_ {\tau} + \beta_ {k} \ell_ {\tau} \quad \left(\| q _ {k} ^ {i} \| _ {\infty} \leq 1 \text {b y i n d u c t i o n h y p o t h e s i s}\right) \\ = \ell_ {\tau}. \\ \end{array}
$$
The induction is complete.
C.2 Analysis of the Policies
Let  $V_{R}:\Delta (\mathcal{A}^{1})\times \Delta (\mathcal{A}^{2})\mapsto \mathbb{R}$  be the regularized Nash gap defined as
$$
V _ {R} (\mu^ {1}, \mu^ {2}) = \sum_ {i = 1, 2} \max _ {\hat {\mu} ^ {i} \in \Delta (\mathcal {A} ^ {i})} \left\{(\hat {\mu} ^ {i} - \mu^ {i}) ^ {\top} R _ {i} \mu^ {- i} + \tau \nu (\hat {\mu} ^ {i}) - \tau \nu (\mu^ {i}) \right\},
$$
where  $\nu (\cdot)$  is the entropy function. A sequence of properties regarding  $V_{R}(\cdot ,\cdot)$  are provided in Lemma C.7. For simplicity of notation, we use  $\nabla_1V_R(\cdot ,\cdot)$  (respectively,  $\nabla_2V_R(\cdot ,\cdot))$  to represent the gradient with respect to the first argument (respectively, the second argument).
Lemma C.3. It holds for all  $k \geq 0$  that
$$
\mathbb {E} [ V _ {R} (\pi_ {k + 1} ^ {1}, \pi_ {k + 1} ^ {2}) ] \leq \left(1 - \frac {\beta_ {k}}{2}\right) \mathbb {E} [ V _ {R} (\pi_ {k} ^ {1}, \pi_ {k} ^ {2}) ] + \frac {\ell_ {\tau} \alpha_ {k}}{4} \sum_ {i = 1, 2} \mathbb {E} [ \| q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \| _ {2} ^ {2} ] + 2 L _ {\tau} \beta_ {k} ^ {2},
$$
where we recall that  $L_{\tau} = \tau /\ell_{\tau} + A_{\mathrm{max}}^{2} / \tau$
Proof of Lemma C.3. Using the smoothness property of  $V_{R}(\cdot, \cdot)$  (cf. Lemma C.7 (1)) and the update equation in Algorithm 1 Line 3, we have for any  $k \geq 0$  that
$$
\begin{array}{l} V _ {R} \left(\pi_ {k + 1} ^ {1}, \pi_ {k + 1} ^ {2}\right) \leq V _ {R} \left(\pi_ {k} ^ {1}, \pi_ {k} ^ {2}\right) + \beta_ {k} \left\langle \nabla_ {2} V _ {R} \left(\pi_ {k} ^ {1}, \pi_ {k} ^ {2}\right), \sigma_ {\tau} \left(q _ {k} ^ {2}\right) - \pi_ {k} ^ {2} \right\rangle \\ + \beta_ {k} \langle \nabla_ {1} V _ {R} (\pi_ {k} ^ {1}, \pi_ {k} ^ {2}), \sigma_ {\tau} (q _ {k} ^ {1}) - \pi_ {k} ^ {1} \rangle + \frac {L _ {\tau} \beta_ {k} ^ {2}}{2} \sum_ {i = 1, 2} \| \sigma_ {\tau} (q _ {k} ^ {i}) - \pi_ {k} ^ {i} \| _ {2} ^ {2} \\ \leq V _ {R} \left(\pi_ {k} ^ {1}, \pi_ {k} ^ {2}\right) + \beta_ {k} \left\langle \nabla_ {2} V _ {R} \left(\pi_ {k} ^ {1}, \pi_ {k} ^ {2}\right), \sigma_ {\tau} \left(R _ {2} \pi_ {k} ^ {1}\right) - \pi_ {k} ^ {2} \right\rangle \\ + \beta_ {k} \langle \nabla_ {1} V _ {R} \left(\pi_ {k} ^ {1}, \pi_ {k + 1} ^ {2}\right), \sigma_ {\tau} \left(R _ {1} \pi_ {k} ^ {2}\right) - \pi_ {k} ^ {1} \rangle \\ + \beta_ {k} \langle \nabla_ {2} V _ {R} (\pi_ {k} ^ {1}, \pi_ {k} ^ {2}), \sigma_ {\tau} (q _ {k} ^ {2}) - \sigma_ {\tau} (R _ {2} \pi_ {k} ^ {1}) \rangle \\ + \beta_ {k} \langle \nabla_ {1} V _ {R} (\pi_ {k} ^ {1}, \pi_ {k + 1} ^ {2}), \sigma_ {\tau} (q _ {k} ^ {1}) - \sigma_ {\tau} (R _ {1} \pi_ {k} ^ {2}) \rangle + 2 L _ {\tau} \beta_ {k} ^ {2} \\ \leq \left(1 - \frac {\beta_ {k}}{2}\right) V _ {R} (\pi_ {k} ^ {1}, \pi_ {k} ^ {2}) + 4 \beta_ {k} \left(\frac {1}{\tau \ell_ {\tau} ^ {2}} + \frac {A _ {\mathrm {m a x}} ^ {2}}{\tau^ {3}}\right) \sum_ {i = 1, 2} \| q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \| _ {2} ^ {2} + 2 L _ {\tau} \beta_ {k} ^ {2}, \\ \end{array}
$$
where the last line follows from Lemma C.7 (2) and (3). Taking expectations on both sides of the previous inequality and using the condition that  $c_{\alpha, \beta} = \frac{\beta_k}{\alpha_k} \leq \min \left( \frac{\tau \ell_{\tau}^3}{32}, \frac{\ell_{\tau} \tau^3}{32A_{\max}^2} \right)$  (cf. Condition 2.1), we have
$$
\mathbb {E} [ V _ {R} (\pi_ {k + 1} ^ {1}, \pi_ {k + 1} ^ {2}) ] \leq \left(1 - \frac {\beta_ {k}}{2}\right) \mathbb {E} [ V _ {R} (\pi_ {k} ^ {1}, \pi_ {k} ^ {2}) ] + \frac {\ell_ {\tau} \alpha_ {k}}{4} \sum_ {i = 1, 2} \mathbb {E} [ \| q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \| _ {2} ^ {2} ] + 2 L _ {\tau} \beta_ {k} ^ {2}.
$$
C.3 Analysis of the  $q$ -Functions
We study the  $q$ -functions generated by Algorithm 1 through a stochastic approximation framework. For  $i \in \{1,2\}$ , let  $F^i: \mathbb{R}^{|\mathcal{A}^i|} \times \mathcal{A}^i \times \mathcal{A}^{-i} \mapsto \mathbb{R}^{|\mathcal{A}^i|}$  be an operator defined as
$$
[ F ^ {i} (q ^ {i}, a _ {0} ^ {i}, a _ {0} ^ {- i}) ] (a ^ {i}) = \mathbb {1} _ {\{a _ {0} ^ {i} = a ^ {i} \}} \left(R _ {i} (a _ {0} ^ {i}, a _ {0} ^ {- i}) - q ^ {i} (a _ {0} ^ {i})\right), \quad \forall \left(q ^ {i}, a _ {0} ^ {i}, a _ {0} ^ {- i}\right) \text {a n d} a ^ {i}.
$$
Then, Algorithm 1 Line 5 can be compactly written as
$$
q _ {k + 1} ^ {i} = q _ {k} ^ {i} + \alpha_ {k} F ^ {i} \left(q _ {k} ^ {i}, A _ {k} ^ {i}, A _ {k} ^ {- i}\right). \tag {24}
$$
Given a joint policy  $(\pi^1,\pi^2)$ , let  $\bar{F}_{\pi}^{i}:\mathbb{R}^{|\mathcal{A}^{i}|}\mapsto \mathbb{R}^{|\mathcal{A}^{i}|}$  be defined as
$$
\bar {F} _ {\pi} ^ {i} \left(q ^ {i}\right) := \mathbb {E} _ {A ^ {i} \sim \pi^ {i} (\cdot), A ^ {- i} \sim \pi^ {- i} (\cdot)} \left[ F ^ {i} \left(q ^ {i}, A ^ {i}, A ^ {- i}\right) \right] = \operatorname {d i a g} \left(\pi^ {i}\right) \left(R _ {i} \pi^ {- i} - q ^ {i}\right).
$$
Then, Eq. (24) can be viewed as a stochastic approximation algorithm for solving the slowly time-varying equation  $\bar{F}_{\pi_k}^i (q^i) = 0$
Lemma C.4. The following inequality holds for all  $k \geq 0$ :
$$
\sum_ {i = 1, 2} \mathbb {E} [ \| q _ {k + 1} ^ {i} - R _ {i} \pi_ {k + 1} ^ {- i} \| _ {2} ^ {2} ] \leq \left(1 - \frac {\ell_ {\tau} \alpha_ {k}}{2}\right) \sum_ {i = 1, 2} \mathbb {E} [ \| q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \| _ {2} ^ {2} ] + \frac {\beta_ {k}}{4} \mathbb {E} [ V _ {R} (\pi_ {k} ^ {1}, \pi_ {k} ^ {2}) ] + 1 6 \alpha_ {k} ^ {2}.
$$
Proof of Lemma C.4. For any  $k\geq 0$  and  $i\in \{1,2\}$  , we have
$$
\begin{array}{l} \left\| q _ {k + 1} ^ {i} - R _ {i} \pi_ {k + 1} ^ {- i} \right\| _ {2} ^ {2} \\ = \| q _ {k + 1} ^ {i} - q _ {k} ^ {i} + q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} + R _ {i} \pi_ {k} ^ {- i} - R _ {i} \pi_ {k + 1} ^ {- i} \| _ {2} ^ {2} \\ = \| q _ {k + 1} ^ {i} - q _ {k} ^ {i} \| _ {2} ^ {2} + \| q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \| _ {2} ^ {2} + \| R _ {i} \pi_ {k} ^ {- i} - R _ {i} \pi_ {k + 1} ^ {- i} \| _ {2} ^ {2} + 2 \langle q _ {k + 1} ^ {i} - q _ {k} ^ {i}, q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \rangle \\ \end{array}
$$
(This follows from the AM-GM inequality, where  $c_{1}, c_{2} > 0$  can be arbitrary.)
$$
\begin{array}{l} = \left(\alpha_ {k} ^ {2} + \frac {\alpha_ {k} \beta_ {k}}{c _ {1}}\right) \| F ^ {i} \left(q _ {k} ^ {i}, A _ {k} ^ {i}, A _ {k} ^ {- i}\right) \| _ {2} ^ {2} + \left(1 + c _ {2} \beta_ {k}\right) \| q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \| _ {2} ^ {2} \\ + 2 \alpha_ {k} \left\langle \bar {F} _ {\pi_ {k}} ^ {i} \left(q _ {k} ^ {i}\right), q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \right\rangle + 2 \alpha_ {k} \left\langle F ^ {i} \left(q _ {k} ^ {i}, A _ {k} ^ {i}, A _ {k} ^ {- i}\right) - \bar {F} _ {\pi_ {k}} ^ {i} \left(q _ {k} ^ {i}\right), q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \right\rangle \\ + \left(\beta_ {k} ^ {2} + \frac {\beta_ {k}}{c _ {2}} + \alpha_ {k} \beta_ {k} c _ {1}\right) \| R _ {i} \| _ {2} ^ {2} \| \sigma_ {\tau} \left(q _ {k} ^ {- i}\right) - \pi_ {k} ^ {- i} \| _ {2} ^ {2}. \\ \end{array}
$$
Taking expectations on both sides of the previous inequality, we have
$$
\begin{array}{l} \mathbb {E} [ \| q _ {k + 1} ^ {i} - R _ {i} \pi_ {k + 1} ^ {- i} \| ^ {2} ] \\ \leq \left(\alpha_ {k} ^ {2} + \frac {\alpha_ {k} \beta_ {k}}{c _ {1}}\right) \mathbb {E} [ \| F ^ {i} (q _ {k} ^ {i}, A _ {k} ^ {i}, A _ {k} ^ {- i}) \| _ {2} ^ {2} ] + (1 + c _ {2} \beta_ {k}) \mathbb {E} [ \| q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \| _ {2} ^ {2} ] \\ + 2 \alpha_ {k} \mathbb {E} [ \langle \bar {F} _ {\pi_ {k}} ^ {i} (q _ {k} ^ {i}), q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \rangle ] + \left(\beta_ {k} ^ {2} + \frac {\beta_ {k}}{c _ {2}} + \alpha_ {k} \beta_ {k} c _ {1}\right) \| R _ {i} \| _ {2} ^ {2} \mathbb {E} [ \| \sigma_ {\tau} (q _ {k} ^ {- i}) - \pi_ {k} ^ {- i} \| _ {2} ^ {2} ], \\ \end{array}
$$
where the term  $\mathbb{E}[\langle F^i (q_k^i,A_k^i,A_k^{-i}) - \bar{F}_{\pi_k}^i (q_k^i),q_k^i -R_i\pi_k^{-i}\rangle ]$  vanishes due to the tower property of conditional expectations. To proceed, observe
$$
\begin{array}{l} \mathbb {E} \left[ \| F ^ {i} \left(q _ {k} ^ {i}, A _ {k} ^ {i}, A _ {k} ^ {- i}\right) \| _ {2} ^ {2} \right] = \mathbb {E} \left[ \sum_ {a ^ {i} \in \mathcal {A} ^ {i}} \mathbb {1} _ {\left\{A _ {k} ^ {i} = a ^ {i} \right\}} \left(R _ {i} \left(A _ {k} ^ {i}, A _ {k} ^ {- i}\right) - q _ {k} ^ {i} \left(A _ {k} ^ {i}\right)\right) ^ {2} \right] \\ = \mathbb {E} \left[ \left(R _ {i} \left(a ^ {i}, A _ {k} ^ {- i}\right) - q _ {k} ^ {i} \left(a ^ {i}\right)\right) ^ {2} \right] \\ \leq \mathbb {E} [ (| R _ {i} (a ^ {i}, A _ {k} ^ {- i}) | + | q _ {k} ^ {i} (a ^ {i}) |) ^ {2} ] \\ \leq 4 \tag {Lemma C.2} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ \langle \bar {F} _ {\pi_ {k}} ^ {i} (q _ {k} ^ {i}), q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \rangle ] = \mathbb {E} [ \langle \operatorname {d i a g} (\pi_ {k} ^ {i}) (R _ {i} \pi_ {k} ^ {- i} - q _ {k} ^ {i}), q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \rangle ] \\ \leq - \ell_ {\tau} \mathbb {E} \left[ \| q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \| _ {2} ^ {2} \right]. \tag {Lemma C.2} \\ \end{array}
$$
In addition, we have
$$
\begin{array}{l} \mathbb {E} [ \| \sigma_ {\tau} \left(q _ {k} ^ {- i}\right) - \pi_ {k} ^ {- i} \| _ {2} ^ {2} ] \leq 2 \mathbb {E} [ \| \sigma_ {\tau} \left(q _ {k} ^ {- i}\right) - \sigma_ {\tau} \left(R _ {- i} \pi_ {k} ^ {i}\right) \| _ {2} ^ {2} ] + 2 \mathbb {E} [ \| \sigma_ {\tau} \left(R _ {- i} \pi_ {k} ^ {i}\right) - \pi_ {k} ^ {- i} \| _ {2} ^ {2} ] \\ (a ^ {2} + b ^ {2} \geq 2 a b) \\ \leq \frac {2}{\tau^ {2}} \mathbb {E} \left[ \left\| q _ {k} ^ {- i} - R _ {- i} \pi_ {k} ^ {i} \right\| _ {2} ^ {2} \right] + \frac {4}{\tau} \mathbb {E} \left[ V _ {R} \left(\pi_ {k} ^ {1}, \pi_ {k} ^ {2}\right) \right], \\ \end{array}
$$
where the last line follows from Lemma C.6. Using the previous 4 inequalities all together, we obtain
$$
\begin{array}{l} \mathbb {E} [ \| q _ {k + 1} ^ {i} - R _ {i} \pi_ {k + 1} ^ {- i} \| _ {2} ^ {2} ] \leq 4 \left(\alpha_ {k} ^ {2} + \frac {\alpha_ {k} \beta_ {k}}{c _ {1}}\right) + (1 - 2 \ell_ {\tau} \alpha_ {k} + c _ {2} \beta_ {k}) \mathbb {E} [ \| q _ {k} ^ {i} - R _ {i} \pi_ {k} ^ {- i} \| _ {2} ^ {2} ] \\ + \frac {4 A _ {\mathrm {m a x}} ^ {2}}{\tau} \left(\beta_ {k} ^ {2} + \frac {\beta_ {k}}{c _ {2}} + \alpha_ {k} \beta_ {k} c _ {1}\right) \mathbb {E} [ V _ {R} (\pi_ {k} ^ {1}, \pi_ {k} ^ {2}) ] \\ + \frac {2 A _ {\operatorname* {m a x}} ^ {2}}{\tau^ {2}} \left(\beta_ {k} ^ {2} + \frac {\beta_ {k}}{c _ {2}} + \alpha_ {k} \beta_ {k} c _ {1}\right) \mathbb {E} [ \| q _ {k} ^ {- i} - R _ {i} \pi_ {k} ^ {- i} \| _ {2} ^ {2} ]. \\ \end{array}
$$
Summing up the previous inequality for  $i \in \{1, 2\}$ , we have