SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {i = 1, 2} \max _ {s \in \mathcal {S}} \left\| \mathcal {T} ^ {i} (v _ {t} ^ {i}) (s) \pi_ {t, K} ^ {- i} (s) - q _ {t, K} ^ {i} (s) \right\| _ {\infty} = \sum_ {i = 1, 2} \left\| \bar {q} _ {k} ^ {i} - q _ {t, K} ^ {i} \right\| _ {\infty} \\ \leq \sum_ {i = 1, 2} \left\| \bar {q} _ {k} ^ {i} - q _ {t, K} ^ {i} \right\| _ {2} \\ \leq \left(2 \sum_ {i = 1, 2} \left\| \bar {q} _ {k} ^ {i} - q _ {t, K} ^ {i} \right\| _ {2} ^ {2}\right) ^ {1 / 2} \quad (a ^ {2} + b ^ {2} \geq 2 a b) \\ \leq 2 \mathcal {L} _ {q} ^ {1 / 2} (t, K). \tag {36} \\ \end{array}
$$
$$
\mathcal {L} _ {v} (t + 1) \leq \gamma \mathcal {L} _ {v} (t) + 4 \mathcal {L} _ {\text {s u m}} (t) + 2 \mathcal {L} _ {q} ^ {1 / 2} (t, K) + 4 \mathcal {L} _ {\pi} (t, K) + 6 \tau \log \left(A _ {\max }\right).
$$
We next establish a one-step Lyapunov drift inequality for  $\mathcal{L}_{\mathrm{sum}}(t)$  in the following lemma.
Lemma D.6. It holds for all  $t \geq 0$  that  $\mathcal{L}_{sum}(t + 1) \leq \gamma \mathcal{L}_{sum}(t) + 2\mathcal{L}_q^{1/2}(t, K)$ .
Proof of Lemma D.6. Using the outer-loop update equation (cf. Algorithm 2 Line 8), we have for any  $t \geq 0$  and  $s \in S$  that
$$
\begin{array}{l} \left| v _ {t + 1} ^ {1} (s) + v _ {t + 1} ^ {2} (s) \right| = \left| \sum_ {i = 1, 2} \pi_ {t, K} ^ {i} (s) ^ {\top} q _ {t, K} ^ {i} (s) \right| \\ \leq \left| \sum_ {i = 1, 2} \pi_ {t, K} ^ {i} (s) ^ {\top} \left(q _ {t, K} ^ {i} (s) - \mathcal {T} ^ {i} \left(v _ {t} ^ {i}\right) (s) \pi_ {t, K} ^ {- i} (s)\right) \right| \\ + \left| \sum_ {i = 1, 2} \pi_ {t, K} ^ {i} (s) \mathcal {T} ^ {i} \left(v _ {t} ^ {i}\right) (s) \pi_ {t, K} ^ {- i} (s) \right| \\ \leq \sum_ {i = 1, 2} \max _ {s \in \mathcal {S}} \| q _ {t, K} ^ {i} (s) - \mathcal {T} ^ {i} (v _ {t} ^ {i}) (s) \pi_ {t, K} ^ {- i} (s) \| _ {\infty} \\ + \sum_ {i = 1, 2} \max _ {(s, a ^ {i}, a ^ {- i})} \left| \mathcal {T} ^ {i} (v _ {t} ^ {i}) (s, a ^ {i}, a ^ {- i}) + \mathcal {T} ^ {- i} (v _ {t} ^ {- i}) (s, a ^ {i}, a ^ {- i}) \right| \\ \leq \sum_ {i = 1, 2} \max _ {s \in \mathcal {S}} \| q _ {t, K} ^ {i} (s) - \mathcal {T} ^ {i} (v _ {t} ^ {i}) (s) \pi_ {t, K} ^ {- i} (s) \| _ {\infty} + \gamma \| v _ {t} ^ {1} + v _ {t} ^ {2} \| _ {\infty}, \\ \end{array}
$$
where the last line follows from the definition of  $\mathcal{T}^i (\cdot)$ . Since the RHS of the previous inequality does not depend on  $s$ , we have
$$
\| v _ {t + 1} ^ {1} + v _ {t + 1} ^ {2} \| _ {\infty} \leq \gamma \| v _ {t} ^ {1} + v _ {t} ^ {2} \| _ {\infty} + \sum_ {i = 1, 2} \max _ {s \in \mathcal {S}} \| q _ {t, K} ^ {i} (s) - \mathcal {T} ^ {i} (v _ {t} ^ {i}) (s) \pi_ {t, K} ^ {- i} (s) \| _ {\infty}.
$$
The result follows from using Eq. (36) to bound the last term on the RHS of the previous inequality and then using  $\mathcal{L}_{\mathrm{sum}}(t)$  and  $\mathcal{L}_q(t,k)$  to simplify the notation.
D.5 Analysis of the Inner Loop
In this section, we establish negative drift inequalities for the Lyapunov functions  $\mathcal{L}_q(t,k)$  and  $\mathcal{L}_{\pi}(t,k)$ , which are defined in terms of the  $q$ -functions and the policies updated in the inner loop of Algorithm 2. For ease of presentation, we write down only the inner loop of Algorithm 2 in Algorithm 3, where we omit the subscript  $t$ , which is used as the index for the outer loop. Similarly, we will write  $\mathcal{L}_q(k)$  for  $\mathcal{L}_q(t,k)$  and  $\mathcal{L}_{\pi}(k)$  for  $\mathcal{L}_{\pi}(t,k)$ . All results derived for the  $q$ -functions and policies of Algorithm 3 can be directly combined with the analysis of outer loop of Algorithm 2 using a conditioning argument together with the Markov property.
Algorithm 3 Inner Loop of Algorithm 2: from Player  $i$ 's Perspective
1: Input: Integer  $K$ , initializations  $q_0^i$  and  $\pi_0^i$ , and a  $v$ -function  $v^i$  from the outer loop. Note that we have  $\| q_0^i \|_{\infty} \leq \frac{1}{1 - \gamma}$ ,  $\| v^i \|_{\infty} \leq \frac{1}{1 - \gamma}$ , and  $\min_{s,a^i} \pi_0^i(a^i \mid s) \geq \ell_\tau$  due to Lemma D.1.
2: for  $k = 0,1,\dots ,K - 1$  do
3:  $\pi_{k + 1}^{i}(s) = \pi_{k}^{i}(s) + \beta_{k}(\sigma_{\tau}(q_{k}^{i}(s)) - \pi_{k}^{i}(s))$  for all  $s\in S$
4: Sample  $A_{k}^{i}\sim \pi_{k + 1}^{i}(\cdot \mid S_{k})$  , and observe  $S_{k + 1}\sim p(\cdot \mid S_k,A_k^i,A_k^{-i})$
5:  $q_{k + 1}^{i}(s,a^{i}) = q_{k}^{i}(s,a^{i}) + \alpha_{k}\mathbb{1}_{\{(S_{k},A_{k}^{i}) = (s,a^{i})\}}\left(R_{i}(S_{k},A_{k}^{i},A_{k}^{-i}) + \gamma v^{i}(S_{k + 1}) - q_{k}^{i}(S_{k},A_{k}^{i})\right)$  for all  $(s,a^i)\in \mathcal{S}\times \mathcal{A}^i$
6: end for
7: Output:  $q_{K}^{i}$  and  $\pi_{K}^{i}$
D.5.1 Analysis of the Policies
We consider  $\{(\pi_k^1,\pi_k^2)\}_{k\geq 0}$  generated by Algorithm 3 and use  $V_{X}(\cdot ,\cdot)$  defined in Appendix D.1 as the Lyapunov function to study them. For simplicity of notation, we use  $\nabla_1V_X(\cdot ,\cdot)$  (respectively,  $\nabla_2V_X(\cdot ,\cdot))$  to denote the gradient with respect to the first argument (respectively, the second argument). Recall that Lemma D.1 implies that  $\pi_k = (\pi_k^1,\pi_k^2)\in \Pi_\tau$  for all  $k\geq 0$ , where  $\Pi_{\tau} = \{(\pi^{1},\pi^{2})\in \Delta (\mathcal{A}^{1})\times \Delta (\mathcal{A}^{2})\mid \min_{a^{1}\in \mathcal{A}^{1}}\pi^{1}(a^{1})\geq \ell_{\tau},\min_{a^{2}\in \mathcal{A}^{2}}\pi^{2}(a^{2})\geq \ell_{\tau}\}$ . The following lemma establishes the properties of  $V_{X}(\cdot ,\cdot)$ .
Lemma D.7. The function  $V_{X}(\cdot ,\cdot)$  has the following properties.
(1) For  $i\in \{1,2\}$ , fix  $\mu^{-i}\in \Delta (\mathcal{A}^{-i})$ , the function  $V_{X}(\mu^{i},\mu^{-i})$  as a function of  $\mu^i$  is  $\tau -\text{strongly}$  convex with respect to  $\| \cdot \| _2$ .
(2)  $V_{X}(\cdot ,\cdot)$  is  $\tilde{L}_{\tau}$  -smooth on  $\Pi_{\tau}$  , where  $\tilde{L}_{\tau} = 2\left(\frac{\tau}{\ell_{\tau}} +\frac{\max(\|X_1\|_2^2,\|X_2\|_2^2)}{\tau} +\| X_1 + X_2^\top \| _2\right).$
(3) It holds for any  $(\mu^1,\mu^2)\in \Delta (\mathcal{A}^1)\times \Delta (\mathcal{A}^2)$  that
$$
\begin{array}{l} \langle \nabla_ {1} V _ {X} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (X _ {1} \mu^ {2}) - \mu^ {1} \rangle + \langle \nabla_ {2} V _ {X} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (X _ {2} \mu^ {1}) - \mu^ {2} \rangle \\ \leq - \frac {7}{8} V _ {X} \left(\mu^ {1}, \mu^ {2}\right) + \frac {1 6}{\tau} \| X _ {1} + X _ {2} ^ {\top} \| _ {2} ^ {2}. \\ \end{array}
$$
(4) For any  $u^1 \in \mathbb{R}^{|\mathcal{A}^1|}$  and  $u^2 \in \mathbb{R}^{|\mathcal{A}^2|}$ , we have for all  $(\mu^1, \mu^2) \in \Pi_\tau$  that
$$
\begin{array}{l} \langle \nabla_ {1} V _ {X} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (u ^ {1}) - \sigma_ {\tau} (X _ {1} \mu^ {2}) \rangle + \langle \nabla_ {2} V _ {X} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (u ^ {2}) - \sigma_ {\tau} (X _ {2} \mu^ {1}) \rangle \\ \leq \frac {1}{8} V _ {X} \left(\mu^ {1}, \mu^ {2}\right) + \frac {8}{\tau} \left(\frac {1}{\ell_ {\tau}} + \frac {\operatorname* {m a x} \left(\| X _ {1} \| _ {2} , \| X _ {2} \| _ {2}\right)}{\tau}\right) ^ {2} \sum_ {i = 1, 2} \| u ^ {i} - X _ {i} \mu^ {- i} \| _ {2} ^ {2}. \\ \end{array}
$$
Proof of Lemma D.7. To begin with, we have by Danskin's theorem [95] that
$$
\nabla_ {1} V _ {X} \left(\mu^ {1}, \mu^ {2}\right) = - \left(X _ {1} + X _ {2} ^ {\top}\right) \mu^ {2} - \tau \nabla \nu \left(\mu^ {1}\right) + X _ {2} ^ {\top} \sigma_ {\tau} \left(X _ {2} \mu^ {1}\right). \tag {37}
$$
Similar result holds for  $\nabla_2 V_X(\mu^1, \mu^2)$ .
(1) It is clear that the function  $V_{X}(\cdot, \cdot)$  is non-negative. The strong convexity follows from the following two observations.
(i) The negative entropy  $-\nu(\cdot)$  is  $1-$  strongly convex with respect to  $\|\cdot\|_2$  [97, Example 5.27].
(ii) Given  $i \in \{1, 2\}$ , the function  $\max_{\hat{\mu}^{-i} \in \Delta(\mathcal{A}^{-i})} \left\{ (\hat{\mu}^{-i})^\top X_{-i} \mu^i + \tau \nu(\hat{\mu}^{-i}) \right\}$  as a function of  $\mu^i$  is the maximum of linear functions in  $\mu^i$ , and therefore is convex.
It follows that, for any  $i \in \{1,2\}$ , the function  $V_{X}(\mu^{1},\mu^{2})$  is  $\tau-$  strongly convex in  $\mu^i$  with respect to  $\| \cdot \|_2$  uniformly for all  $\mu^{-i}$ .
(2) For any  $(\mu^1,\mu^2),(\bar{\mu}^1,\bar{\mu}^2)\in \Pi_\tau$  , we have by Eq. (37) that
$$
\begin{array}{l} \left\| \nabla_ {1} V _ {X} \left(\mu^ {1}, \mu^ {2}\right) - \nabla_ {1} V _ {X} \left(\bar {\mu} ^ {1}, \bar {\mu} ^ {2}\right) \right\| _ {2} \\ = \left\| (X _ {1} + X _ {2} ^ {\top}) (\mu^ {2} - \bar {\mu} ^ {2}) + \tau (\nabla \nu (\mu^ {1}) - \nabla \nu (\bar {\mu} ^ {1})) + X _ {2} ^ {\top} (\sigma_ {\tau} (X _ {2} \bar {\mu} ^ {1}) - \sigma_ {\tau} (X _ {2} \mu^ {1})) \right\| _ {2} \\ \leq \left\| X _ {1} + X _ {2} ^ {\top} \right\| _ {2} \left\| \mu^ {2} - \bar {\mu} ^ {2} \right\| _ {2} + \left(\frac {\tau}{\ell_ {\tau}} + \frac {\left\| X _ {2} \right\| _ {2} ^ {2}}{\tau}\right) \left\| \bar {\mu} ^ {1} - \mu^ {1} \right\| _ {2} \tag {38} \\ \end{array}
$$
where Eq. (38) follows from Lemma C.5 and the Lipschitz continuity of  $\sigma_{\tau}(\cdot)$  [96]. Similarly, we also have
$$
\begin{array}{l} \left\| \nabla_ {2} V _ {X} \left(\mu^ {1}, \mu^ {2}\right) - \nabla_ {2} V _ {X} \left(\bar {\mu} ^ {1}, \bar {\mu} ^ {2}\right) \right\| _ {2} \\ \leq \| X _ {2} + X _ {1} ^ {\top} \| _ {2} \| \mu^ {1} - \bar {\mu} ^ {1} \| _ {2} + \left(\frac {\tau}{\ell_ {\tau}} + \frac {\| X _ {1} \| _ {2} ^ {2}}{\tau}\right) \| \bar {\mu} ^ {2} - \mu^ {2} \| _ {2}. \\ \end{array}
$$
Using the previous 2 inequalities, we have the following result for the full gradient of  $V_{X}(\cdot ,\cdot)$ :
$$
\begin{array}{l} \left\| \nabla V _ {X} \left(\mu^ {1}, \mu^ {2}\right) - \nabla V _ {X} \left(\bar {\mu} ^ {1}, \bar {\mu} ^ {2}\right) \right\| _ {2} ^ {2} \\ = \left\| \nabla_ {1} V _ {X} \left(\mu^ {1}, \mu^ {2}\right) - \nabla_ {1} V _ {X} \left(\bar {\mu} ^ {1}, \bar {\mu} ^ {2}\right) \right\| _ {2} ^ {2} + \left\| \nabla_ {2} V _ {X} \left(\mu^ {1}, \mu^ {2}\right) - \nabla_ {2} V _ {X} \left(\bar {\mu} ^ {1}, \bar {\mu} ^ {2}\right) \right\| _ {2} ^ {2} \\ \leq \sum_ {i = 1, 2} \left[ 2 \left(\frac {\tau}{\ell_ {\tau}} + \frac {\| X _ {- i} \| _ {2} ^ {2}}{\tau}\right) ^ {2} \| \bar {\mu} ^ {i} - \mu^ {i} \| _ {2} ^ {2} + 2 \| X _ {i} + X _ {- i} ^ {\top} \| _ {2} ^ {2} \| \mu^ {- i} - \bar {\mu} ^ {- i} \| _ {2} ^ {2} \right] \quad ((a + b) ^ {2} \leq 2 a ^ {2} + 2 b ^ {2}) \\ \leq 2 \left[ \left(\frac {\tau}{\ell_ {\tau}} + \frac {\operatorname* {m a x} (\| X _ {1} \| _ {2} ^ {2} , \| X _ {2} \| _ {2} ^ {2})}{\tau}\right) ^ {2} + \| X _ {1} + X _ {2} ^ {\top} \| _ {2} ^ {2} \right] \sum_ {i = 1, 2} \| \bar {\mu} ^ {i} - \mu^ {i} \| _ {2} ^ {2}. \\ \end{array}
$$
The previous inequality implies that  $V_{X}(\cdot, \cdot)$  is an  $\tilde{L}_{\tau}$ -smooth function on  $\Pi_{\tau}$  [97], where
$$
\tilde {L} _ {\tau} = 2 \left(\frac {\tau}{\ell_ {\tau}} + \frac {\operatorname* {m a x} (\| X _ {1} \| _ {2} ^ {2} , \| X _ {2} \| _ {2} ^ {2})}{\tau} + \| X _ {1} + X _ {2} ^ {\top} \| _ {2}\right).
$$
(3) Using the formula for the gradient of  $V_{X}(\cdot ,\cdot)$  in Eq. (37), we have
$$
\begin{array}{l} \left\langle \nabla_ {1} V _ {X} \left(\mu^ {1}, \mu^ {2}\right), \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) - \mu^ {1} \right\rangle \\ = \left\langle - \left(X _ {1} + X _ {2} ^ {\top}\right) \mu^ {2} - \tau \nabla \nu \left(\mu^ {1}\right) + X _ {2} ^ {\top} \sigma_ {\tau} \left(X _ {2} \mu^ {1}\right), \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) - \mu^ {1} \right\rangle \\ = \left\langle - \left(X _ {1} + X _ {2} ^ {\top}\right) \mu^ {2} - \tau \nabla \nu \left(\mu^ {1}\right) + X _ {2} ^ {\top} \sigma_ {\tau} \left(X _ {2} \mu^ {1}\right), \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) - \mu^ {1} \right\rangle \\ + \left\langle X _ {1} \mu^ {2} + \tau \nabla \nu \left(\sigma_ {\tau} \left(X _ {1} \mu^ {2}\right)\right), \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) - \mu^ {1} \right\rangle \tag {39} \\ = \tau \left\langle \nabla \nu \left(\sigma_ {\tau} \left(X _ {1} \mu^ {2}\right)\right) - \nabla \nu \left(\mu^ {1}\right), \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) - \mu^ {1} \right\rangle \\ + \left(\sigma_ {\tau} \left(X _ {2} \mu^ {1}\right) - \mu^ {2}\right) ^ {\top} X _ {2} \left(\sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) - \mu^ {1}\right), \\ \end{array}
$$
where Eq. (39) is due to the first order optimality condition  $X_{1}\mu^{2} + \tau \nabla \nu (\sigma_{\tau}(X_1\mu^2)) = 0$ . To proceed, observe that the concavity of  $\nu (\cdot)$  and the optimality condition  $X_{1}\mu^{2} + \tau \nabla \nu (\sigma_{\tau}(X_{1}\mu^{2})) = 0$  together imply that
$$
\begin{array}{l} \left\langle \nabla \nu \left(\sigma_ {\tau} \left(X _ {1} \mu^ {2}\right)\right) - \nabla \nu \left(\mu^ {1}\right), \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) - \mu^ {1} \right\rangle \\ = \left\langle \nabla \nu \left(\mu^ {1}\right) - \nabla \nu \left(\sigma_ {\tau} \left(X _ {1} \mu^ {2}\right)\right), \mu^ {1} - \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) \right\rangle \\ = \left\langle \nabla \nu \left(\mu^ {1}\right), \mu^ {1} - \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) \right\rangle - \left\langle \nabla \nu \left(\sigma_ {\tau} \left(X _ {1} \mu^ {2}\right)\right), \mu^ {1} - \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) \right\rangle \\ \leq \nu (\mu^ {1}) - \nu \left(\sigma_ {\tau} \left(X _ {1} \mu^ {2}\right)\right) - \left\langle \nabla \nu \left(\sigma_ {\tau} \left(X _ {1} \mu^ {2}\right)\right), \mu^ {1} - \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) \right\rangle \quad (\text {C o n c a v i t y} \nu (\cdot)) \\ = \nu (\mu^ {1}) - \nu \left(\sigma_ {\tau} \left(X _ {1} \mu^ {2}\right)\right) + \frac {1}{\tau} \langle X _ {1} \mu^ {2}, \mu^ {1} - \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) \rangle \\ = \frac {1}{\tau} \left[ \left(\mu^ {1}\right) ^ {\top} X _ {1} \mu^ {2} + \tau \nu (\mu^ {1}) - \max  _ {\hat {\mu} ^ {1} \in \Delta \left(\mathcal {A} ^ {1}\right)} \left\{\left(\hat {\mu} ^ {1}\right) ^ {\top} X _ {1} \mu^ {2} + \tau \nu \left(\hat {\mu} ^ {1}\right) \right\} \right]. \\ \end{array}
$$
Therefore, we have from the previous 2 inequalities that
$$
\langle \nabla_ {1} V _ {X} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (X _ {1} \mu^ {2}) - \mu^ {1} \rangle
$$
$$
\begin{array}{l} \leq (\mu^ {1}) ^ {\top} X _ {1} \mu^ {2} + \tau \nu (\mu^ {1}) - \max  _ {\hat {\mu} ^ {1} \in \Delta (\mathcal {A} ^ {1})} \left\{(\hat {\mu} ^ {1}) ^ {\top} X _ {1} \mu^ {2} + \tau \nu (\hat {\mu} ^ {1}) \right\} \\ + \left(\sigma_ {\tau} \left(X _ {2} \mu^ {1}\right) - \mu^ {2}\right) ^ {\top} X _ {2} \left(\sigma_ {\tau} \left(X _ {i} \mu^ {2}\right) - \mu^ {1}\right). \\ \end{array}
$$
Similarly, we also have
$$
\begin{array}{l} \langle \nabla_ {2} V _ {X} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (X _ {2} \mu^ {1}) - \mu^ {2} \rangle \\ \leq (\mu^ {2}) ^ {\top} X _ {2} \mu^ {1} + \tau \nu (\mu^ {2}) - \max  _ {\hat {\mu} ^ {2} \in \Delta (\mathcal {A} ^ {2})} \left\{(\hat {\mu} ^ {2}) ^ {\top} X _ {2} \mu^ {1} + \tau \nu (\hat {\mu} ^ {2}) \right\} \\ + \left(\sigma_ {\tau} (X _ {1} \mu^ {2}) - \mu^ {1}\right) ^ {\top} X _ {1} (\sigma_ {\tau} (X _ {2} \mu^ {1}) - \mu^ {2}). \\ \end{array}
$$
Adding up the previous 2 inequalities, we obtain
$$
\begin{array}{l} \langle \nabla_ {1} V _ {X} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (X _ {1} \mu^ {2}) - \mu^ {1} \rangle + \langle \nabla_ {2} V _ {X} (\mu^ {1}, \mu^ {2}), \sigma_ {\tau} (X _ {2} \mu^ {1}) - \mu^ {2} \rangle \\ \leq - V _ {X} (\mu^ {1}, \mu^ {2}) + (\sigma_ {\tau} (X _ {1} \mu^ {2}) - \mu^ {1}) ^ {\top} (X _ {1} + X _ {2} ^ {\top}) (\sigma_ {\tau} (X _ {2} \mu^ {1}) - \mu^ {2}) \\ \leq - V _ {X} \left(\mu^ {1}, \mu^ {2}\right) + \left\| \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) - \mu^ {1} \right\| _ {2} \left\| X _ {1} + X _ {2} ^ {\top} \right\| _ {2} \left\| \sigma_ {\tau} \left(X _ {2} \mu^ {1}\right) - \mu^ {2} \right\| _ {2} \\ \leq - V _ {X} \left(\mu^ {1}, \mu^ {2}\right) + 2 \| \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) - \mu^ {1} \| _ {2} \| X _ {1} + X _ {2} ^ {\top} \| _ {2}, \tag {40} \\ \end{array}
$$
where the last line follows from  $\| \sigma_{\tau}(X_2\mu^1) - \mu^2 \|_2 \leq \| \sigma_{\tau}(X_2\mu^1) \|_1 + \| \mu^2 \|_1 \leq 2$ . Using Lemma D.7 (1) together with the quadratic growth property of strongly convex functions, we have
$$
\left\| \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) - \mu^ {1} \right\| _ {2} \leq \frac {\sqrt {2}}{\sqrt {\tau}} V _ {X} \left(\mu^ {1}, \mu^ {2}\right) ^ {1 / 2}.
$$
$$
\begin{array}{l} \left\langle \nabla_ {1} V _ {X} \left(\mu^ {1}, \mu^ {2}\right), \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) - \mu^ {1} \right\rangle + \left\langle \nabla_ {2} V _ {X} \left(\mu^ {1}, \mu^ {2}\right), \sigma_ {\tau} \left(X _ {2} \mu^ {1}\right) - \mu^ {2} \right\rangle \\ \leq - V _ {X} \left(\mu^ {1}, \mu^ {2}\right) + 2 \left\| \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) - \mu^ {1}\right) \| _ {2} \| X _ {1} + X _ {2} ^ {\top} \| _ {2} \\ \leq - V _ {X} \left(\mu^ {1}, \mu^ {2}\right) + \frac {2 \sqrt {2}}{\sqrt {\tau}} V _ {X} \left(\mu^ {1}, \mu^ {2}\right) ^ {1 / 2} \| X _ {1} + X _ {2} ^ {\top} \| _ {2} \\ \leq - \frac {7}{8} V _ {X} \left(\mu^ {1}, \mu^ {2}\right) + \frac {1 6}{\tau} \| X _ {1} + X _ {2} ^ {\top} \| _ {2} ^ {2}, \\ \end{array}
$$
where the last line follows from  $a^2 + b^2 \geq 2ab$ .
(4) For any  $u^1 \in \mathbb{R}^{|\mathcal{A}^1|}$ , using the formula of the gradient of  $V_X(\cdot, \cdot)$  from Eq. (37), we have
$$
\begin{array}{l} \left\langle \nabla_ {1} V _ {X} \left(\mu^ {1}, \mu^ {2}\right), \sigma_ {\tau} \left(u ^ {1}\right) - \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) \right\rangle \\ = \left\langle - \left(X _ {1} + X _ {2} ^ {\top}\right) \mu^ {2} - \tau \nabla \nu (\mu^ {1}) + X _ {2} ^ {\top} \sigma_ {\tau} \left(X _ {2} \mu^ {1}\right), \sigma_ {\tau} (u ^ {1}) - \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) \right\rangle \\ = \left\langle - \left(X _ {1} + X _ {2} ^ {\top}\right) \mu^ {2} - \tau \nabla \nu \left(\mu^ {1}\right) + X _ {2} ^ {\top} \sigma_ {\tau} \left(X _ {2} \mu^ {1}\right), \sigma_ {\tau} \left(u ^ {1}\right) - \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) \right\rangle \\ + \left\langle X _ {1} \mu^ {2} + \tau \nabla \nu \left(\sigma_ {\tau} \left(X _ {1} \mu^ {2}\right)\right), \sigma_ {\tau} \left(u ^ {1}\right) - \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) \right\rangle \quad (\text {F i r s t o r d e r o p e r t i a l i t y c o n d i t i o n}) \\ = \tau \left\langle \nabla \nu \left(\sigma_ {\tau} \left(X _ {1} \mu^ {2}\right)\right) - \nabla \nu \left(\mu^ {1}\right), \sigma_ {\tau} \left(u ^ {1}\right) - \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) \right\rangle \\ + \left(\sigma_ {\tau} \left(X _ {2} \mu^ {1}\right) - \mu^ {2}\right) ^ {\top} X _ {2} \left(\sigma_ {\tau} \left(u ^ {1}\right) - \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right)\right) \\ \leq \left(\tau \| \nabla \nu \left(\sigma_ {\tau} \left(X _ {1} \mu^ {2}\right)\right) - \nabla \nu \left(\mu^ {1}\right) \| _ {2} + \| \sigma_ {\tau} \left(X _ {2} \mu^ {1}\right) - \mu^ {2} \| _ {2} \| X _ {2} \| _ {2}\right) \| \sigma_ {\tau} (u ^ {1}) - \sigma_ {\tau} \left(X _ {1} \mu^ {2}\right) \| _ {2} \\ \leq \left(\frac {\tau}{\ell_ {\tau}} \| \sigma_ {\tau} (X _ {1} \mu^ {2}) - \mu^ {1} \| _ {2} + \| \sigma_ {\tau} (X _ {2} \mu^ {1}) - \mu^ {2} \| _ {2} \| X _ {2} \| _ {2}\right) \frac {1}{\tau} \| u ^ {1} - X _ {1} \mu^ {2} \| _ {2} \\ \leq \frac {\sqrt {2}}{\sqrt {\tau}} \left(\frac {1}{\ell_ {\tau}} + \frac {\| X _ {2} \| _ {2}}{\tau}\right) V _ {X} \left(\mu^ {1}, \mu^ {2}\right) ^ {1 / 2} \| u ^ {1} - X _ {1} \mu^ {2} \| _ {2} \\ \leq \frac {1}{1 6} V _ {X} (\mu^ {1}, \mu^ {2}) + \frac {8}{\tau} \left(\frac {1}{\ell_ {\tau}} + \frac {\| X _ {2} \| _ {2}}{\tau}\right) ^ {2} \| u ^ {1} - X _ {1} \mu^ {2} \| _ {2} ^ {2}, \\ \end{array}
$$
where the third last inequality follows from the  $\frac{1}{\ell_{\tau}}$ -smoothness of  $\nu(\cdot)$  on  $\Pi_{\tau}$ , the second last inequality follows from Lemma D.7 (1) together with the quadratic growth property of strongly
convex functions, and the last inequality follows from  $a^2 + b^2 \geq 2ab$ . Similarly, we also have for any  $u^2 \in \mathbb{R}^{|\mathcal{A}^2|}$  that
$$
\begin{array}{l} \left\langle \nabla_ {2} V _ {X} \left(\mu^ {1}, \mu^ {2}\right), \sigma_ {\tau} \left(u ^ {2}\right) - \sigma_ {\tau} \left(X _ {2} \mu^ {1}\right) \right\rangle \\ \leq \frac {1}{1 6} V _ {X} (\mu^ {1}, \mu^ {2}) + \frac {8}{\tau} \left(\frac {1}{\ell_ {\tau}} + \frac {\| X _ {1} \| _ {2}}{\tau}\right) ^ {2} \| u ^ {2} - X _ {2} \mu^ {1} \| _ {2} ^ {2}. \\ \end{array}
$$
With the properties of  $V_{X}(\cdot ,\cdot)$  established in Lemma D.7, we now use it as a Lyapunov function to study  $(\pi_k^1,\pi_k^2)$  generated by Algorithm 3. Specifically, using the smoothness of  $V_{X}(\cdot ,\cdot)$  (cf. Lemma D.7 (2)), the update equation in Algorithm 3 Line 3, and Lemma D.7 (3) and (4), we have the desired one-step Lyapunov drift inequality for  $\mathcal{L}_{\pi}(k)$ , which is presented in the following.
Lemma D.8. The following inequality holds for all  $k \geq 0$ :
$$
\begin{array}{l} \mathcal {L} _ {\pi} (k + 1) \leq \left(1 - \frac {3 \beta_ {k}}{4}\right) \mathcal {L} _ {\pi} (k) + \frac {1 6 A _ {\max } ^ {2} \beta_ {k}}{\tau} \| v ^ {1} + v ^ {2} \| _ {\infty} ^ {2} \\ + \frac {3 2 A _ {\operatorname* {m a x}} ^ {2} \beta_ {k}}{\tau^ {3} \ell_ {\tau} ^ {2} (1 - \gamma) ^ {2}} \mathcal {L} _ {q} (k) + 2 L _ {\tau} \beta_ {k} ^ {2}. \\ \end{array}
$$
Proof of Lemma D.8. We will use  $V_{v,s}(\cdot, \cdot)$  (see Lemma D.1) as the Lyapunov function to study the evolution of  $(\pi_k^1(s), \pi_k^2(s))$ . To begin with, we identify the smoothness parameter of  $V_{v,s}(\cdot, \cdot)$ . Using Lemma D.7 (1) and the definition of  $V_{v,s}(\cdot, \cdot)$ , we have
$$
\tilde {L} _ {\tau} = 2 \left(\frac {\tau}{\ell_ {\tau}} + \frac {\operatorname* {m a x} \left(\left\| X _ {1} \right\| _ {2} ^ {2} , \left\| X _ {2} \right\| _ {2} ^ {2}\right)}{\tau} + \left\| X _ {1} + X _ {2} ^ {\top} \right\| _ {2}\right)
$$
$$
= 2 \left(\frac {\tau}{\ell_ {\tau}} + \frac {\operatorname* {m a x} (\| \mathcal {T} ^ {1} (v ^ {1}) (s) \| _ {2} ^ {2} , \| \mathcal {T} ^ {2} (v ^ {2}) (s) \| _ {2} ^ {2})}{\tau} + \| \mathcal {T} ^ {1} (v ^ {1}) (s) + \mathcal {T} ^ {2} (v ^ {2}) (s) ^ {\top} \| _ {2}\right)
$$
$$
\leq 2 \left(\frac {\tau}{\ell_ {\tau}} + \frac {A _ {\operatorname* {m a x}} ^ {2}}{\tau (1 - \gamma) ^ {2}} + \frac {2 A _ {\operatorname* {m a x}}}{1 - \gamma}\right)
$$
(This follows from  $|\mathcal{T}^i (v^i)(s,a^i,a^{-i})|\leq \frac{1}{1 - \gamma},\forall (s,a^i,a^{-i})$  and  $i\in \{1,2\}$
$$
:= L _ {\tau}.
$$
Therefore,  $V_{v,s}(\cdot ,\cdot)$  is an  $L_{\tau}$ -smooth function on  $\Pi_{\tau}$ . Using the smoothness of  $V_{v,s}(\cdot ,\cdot)$ , for any  $s\in S$ , we have by the policy update equation (cf. Algorithm 3 Line 3) that