SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} V _ {v, s} \left(\pi_ {k + 1} ^ {1} (s), \pi_ {k + 1} ^ {2} (s)\right) \\ \leq V _ {v, s} \left(\pi_ {k} ^ {1} (s), \pi_ {k} ^ {2} (s)\right) + \beta_ {k} \left\langle \nabla_ {2} V _ {v, s} \left(\pi_ {k} ^ {1} (s), \pi_ {k} ^ {2} (s)\right), \sigma_ {\tau} \left(q _ {k} ^ {2} (s)\right) - \pi_ {k} ^ {2} (s) \right\rangle \\ + \beta_ {k} \langle \nabla_ {1} V _ {v, s} (\pi_ {k} ^ {1} (s), \pi_ {k} ^ {2} (s)), \sigma_ {\tau} (q _ {k} ^ {1} (s)) - \pi_ {k} ^ {1} (s) \rangle + \frac {L _ {\tau} \beta_ {k} ^ {2}}{2} \sum_ {i = 1, 2} \| \sigma_ {\tau} (q _ {k} ^ {1} (s)) - \pi_ {k} ^ {1} (s) \| _ {2} ^ {2} \\ \leq V _ {v, s} \left(\pi_ {k} ^ {1} (s), \pi_ {k} ^ {2} (s)\right) + \beta_ {k} \left\langle \nabla_ {2} V _ {v, s} \left(\pi_ {k} ^ {1} (s), \pi_ {k} ^ {2} (s)\right), \sigma_ {\tau} \left(\mathcal {T} ^ {2} (v ^ {2}) (s) \pi_ {k} ^ {1} (s)\right) - \pi_ {k} ^ {2} (s) \right\rangle \\ + \beta_ {k} \langle \nabla_ {1} V _ {v, s} \left(\pi_ {k} ^ {1} (s), \pi_ {k + 1} ^ {2} (s)\right), \sigma_ {\tau} \left(\mathcal {T} ^ {1} \left(v ^ {1}\right) (s) \pi_ {k} ^ {2} (s)\right) - \pi_ {k} ^ {1} (s) \rangle \\ + \beta_ {k} \langle \nabla_ {2} V _ {v, s} \left(\pi_ {k} ^ {1} (s), \pi_ {k} ^ {2} (s)\right), \sigma_ {\tau} \left(q _ {k} ^ {2} (s)\right) - \sigma_ {\tau} \left(\mathcal {T} ^ {2} \left(v ^ {2}\right) (s) \pi_ {k} ^ {1} (s)\right) \rangle \\ + \beta_ {k} \langle \nabla_ {1} V _ {v, s} \left(\pi_ {k} ^ {1} (s), \pi_ {k + 1} ^ {2} (s)\right), \sigma_ {\tau} \left(q _ {k} ^ {1} (s)\right) - \sigma_ {\tau} \left(\mathcal {T} ^ {1} \left(v ^ {1}\right) \left(s\right) \pi_ {k} ^ {2} (s)\right) \rangle + 2 L _ {\tau} \beta_ {k} ^ {2} \\ \leq \left(1 - \frac {3 \beta_ {k}}{4}\right) V _ {v, s} \left(\pi_ {k} ^ {1} (s), \pi_ {k} ^ {2} (s)\right) + \frac {1 6 \beta_ {k}}{\tau} \| \mathcal {T} ^ {1} \left(v ^ {1}\right) (s) + \mathcal {T} ^ {2} \left(v ^ {2}\right) (s) ^ {\top} \| _ {2} ^ {2} \\ \end{array}
$$
$$
+ \frac {8 \beta_ {k}}{\tau} \left(\frac {1}{\ell_ {\tau}} + \frac {\max _ {i \in \{1 , 2 \}} \| \mathcal {T} ^ {i} (v ^ {i}) (s) \| _ {2}}{\tau}\right) ^ {2} \sum_ {i = 1, 2} \| q _ {k} ^ {i} (s) - \mathcal {T} ^ {i} (v ^ {i}) (s) \pi_ {k} ^ {2} (s) \| _ {2} ^ {2} + 2 L _ {\tau} \beta_ {k} ^ {2}.
$$
where the last line follows from Lemma D.7 (3) and (4). Since  $\max_{i\in \{1,2\}}\| \mathcal{T}^i (v^i)(s)\| _2\leq \frac{A_{\max}}{1 - \gamma}$  and
$$
\left\| \mathcal {T} ^ {1} (v ^ {1}) (s) + \mathcal {T} ^ {2} (v ^ {2}) (s) ^ {\top} \right\| _ {2} ^ {2} \leq A _ {\max } ^ {2} \| v ^ {1} + v ^ {2} \| _ {\infty} ^ {2},
$$
$$
\begin{array}{l} V _ {v, s} (\pi_ {k + 1} ^ {1} (s), \pi_ {k + 1} ^ {2} (s)) \\ \leq \left(1 - \frac {3 \beta_ {k}}{4}\right) V _ {v, s} (\pi_ {k} ^ {1} (s), \pi_ {k} ^ {2} (s)) + \frac {1 6 \beta_ {k} A _ {\mathrm {m a x}} ^ {2}}{\tau} \| v ^ {1} + v ^ {2} \| _ {\infty} ^ {2} \\ + \frac {8 \beta_ {k}}{\tau} \left(\frac {1}{\ell_ {\tau}} + \frac {A _ {\mathrm {m a x}}}{\tau (1 - \gamma)}\right) ^ {2} \sum_ {i = 1, 2} \| q _ {k} ^ {i} (s) - \mathcal {T} ^ {i} (v ^ {i}) (s) \pi_ {k} ^ {2} (s) \| _ {2} ^ {2} + 2 L _ {\tau} \beta_ {k} ^ {2} \\ \leq \left(1 - \frac {3 \beta_ {k}}{4}\right) \max _ {s \in \mathcal {S}} V _ {v, s} (\pi_ {k} ^ {1} (s), \pi_ {k} ^ {2} (s)) + \frac {1 6 \beta_ {k} A _ {\operatorname* {m a x}} ^ {2}}{\tau} \| v ^ {1} + v ^ {2} \| _ {\infty} ^ {2} \\ + \frac {8 \beta_ {k}}{\tau} \left(\frac {1}{\ell_ {\tau}} + \frac {A _ {\mathrm {m a x}}}{\tau (1 - \gamma)}\right) ^ {2} \sum_ {i = 1, 2} \sum_ {s \in S} \| q _ {k} ^ {i} (s) - \mathcal {T} ^ {i} (v ^ {i}) (s) \pi_ {k} ^ {2} (s) \| _ {2} ^ {2} + 2 L _ {\tau} \beta_ {k} ^ {2} \\ = \left(1 - \frac {3 \beta_ {k}}{4}\right) \mathcal {L} _ {\pi} (k) + \frac {1 6 \beta_ {k} A _ {\mathrm {m a x}} ^ {2}}{\tau} \| v ^ {1} + v ^ {2} \| _ {\infty} ^ {2} + \frac {8 \beta_ {k}}{\tau} \left(\frac {1}{\ell_ {\tau}} + \frac {A _ {\mathrm {m a x}}}{\tau (1 - \gamma)}\right) ^ {2} \mathcal {L} _ {q} (k) + 2 L _ {\tau} \beta_ {k} ^ {2}. \\ \end{array}
$$
Since the RHS of the previous inequality does not depend on  $s$ , we have
$$
\begin{array}{l} \mathcal {L} _ {\pi} (k + 1) \leq \left(1 - \frac {3 \beta_ {k}}{4}\right) \mathcal {L} _ {\pi} (k) + \frac {1 6 \beta_ {k} A _ {\mathrm {m a x}} ^ {2}}{\tau} \| v ^ {1} + v ^ {2} \| _ {\infty} ^ {2} \\ + \frac {8 \beta_ {k}}{\tau} \left(\frac {1}{\ell_ {\tau}} + \frac {A _ {\mathrm {m a x}}}{\tau (1 - \gamma)}\right) ^ {2} \mathcal {L} _ {q} (k) + 2 L _ {\tau} \beta_ {k} ^ {2} \\ \leq \left(1 - \frac {3 \beta_ {k}}{4}\right) \mathcal {L} _ {\pi} (k) + \frac {1 6 \beta_ {k} A _ {\operatorname* {m a x}} ^ {2}}{\tau} \| v ^ {1} + v ^ {2} \| _ {\infty} ^ {2} \\ + \frac {3 2 A _ {\mathrm {m a x}} ^ {2} \beta_ {k}}{\tau^ {3} \ell_ {\tau} ^ {2} (1 - \gamma) ^ {2}} \mathcal {L} _ {q} (k) + 2 L _ {\tau} \beta_ {k} ^ {2}, \\ \end{array}
$$
where the last line follows from  $\tau \leq 1 / (1 - \gamma)$
D.5.2 Analysis of the  $q$ -Functions
In this section, we consider  $q_{k}^{i}$  generated by Algorithm 3. We begin by reformulating the update of the  $q$ -function as a stochastic approximation algorithm for estimating a time-varying target. For  $i \in \{1,2\}$ , fixing  $v^{i} \in \mathbb{R}^{|S|}$ , let  $F^{i}: \mathbb{R}^{|S||A^{i}|} \times S \times A^{i} \times A^{-i} \times S \mapsto \mathbb{R}^{|S||A^{i}|}$  be defined as
$$
[ F ^ {i} (q ^ {i}, s _ {0}, a _ {0} ^ {i}, a _ {0} ^ {- i}, s _ {1}) ] (s, a ^ {i}) = \mathbb {1} _ {\{(s, a ^ {i}) = (s _ {0}, a _ {0} ^ {i}) \}} \left(R _ {i} (s _ {0}, a _ {0} ^ {i}, a _ {0} ^ {- i}) + \gamma v ^ {i} (s _ {1}) - q ^ {i} (s _ {0}, a _ {0} ^ {i})\right)
$$
for all  $(q^i, s_0, a_0^i, a_0^{-i}, s_1)$  and  $(s, a^i)$ . Then Algorithm 3 Line 5 can be compactly written as
$$
q _ {k + 1} ^ {i} = q _ {k} ^ {i} + \alpha_ {k} F ^ {i} \left(q _ {k} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right). \tag {41}
$$
Denote the stationary distribution of the Markov chain  $\{S_k\}$  induced by the joint policy  $\pi_{k} = (\pi_{k}^{1},\pi_{k}^{2})$  by  $\mu_{k}\in \Delta (S)$ , the existence and uniqueness of which is guaranteed by Lemma D.1 and Lemma B.1 (1). Let  $\bar{F}_k^i:\mathbb{R}^{|\mathcal{S}||\mathcal{A}^i|}\mapsto \mathbb{R}^{|\mathcal{S}||\mathcal{A}^i|}$  be defined as
$$
\bar {F} _ {k} ^ {i} (q ^ {i}) = \mathbb {E} _ {S _ {0} \sim \mu_ {k} (\cdot), A _ {0} ^ {i} \sim \pi_ {k} ^ {i} (\cdot | S _ {0}), A _ {0} ^ {- i} \sim \pi_ {k} ^ {- i} (\cdot | S _ {0}), S _ {1} \sim p (\cdot | S _ {0}, A _ {0} ^ {i}, A _ {0} ^ {- i})} \left[ F ^ {i} (q ^ {i}, S _ {0}, A _ {0} ^ {i}, A _ {0} ^ {- i}, S _ {1}) \right]
$$
for all  $q^i \in \mathbb{R}^{|\mathcal{S}| |\mathcal{A}^i|}$ . Then, Eq. (41) can be viewed as a stochastic approximation algorithm for solving the (time-varying) equation  $\bar{F}_k^i(q^i) = 0$  with time-inhomogeneous Markovian noise  $\{(S_k, A_k^i, A_k^{-i}, S_{k+1})\}_{k \geq 0}$ . We next establish the properties of the operators  $F^i(\cdot)$  and  $\bar{F}_k^i(\cdot)$  in the following lemma.
Lemma D.9. The following properties hold for  $i \in \{1,2\}$ .
(1) It holds that  $\| F^i(q_1^i, s_0, a_0^i, a_0^{-i}, s_1) - F^i(q_2^i, s_0, a_0^i, a_0^{-i}, s_1) \|_2 \leq \| q_1^i - q_2^i \|_2$  for any  $(q_1^i, q_2^i)$  and  $(s_0, a_0^i, a_0^{-i}, s_1)$ .
(2) It holds that  $\| F^i (0,s_0,a_0^i,a_0^{-i},s_1)\| _2\leq \frac{1}{1 - \gamma}$  for all  $(s_0,a_0^i,a_0^{-i},s_1)$
(3)  $\bar{F}_k^i (q^i) = 0$  has a unique solution  $\bar{q}_k^i$ , which is given as  $\bar{q}_k^i (s) = \mathcal{T}^i (v^i)(s)\pi_k^{-i}(s)$  for all  $s$ .
(4) It holds that  $\langle \bar{F}_k^i (q_1^i) - \bar{F}_k^i (q_2^i),q_1^i -q_2^i\rangle \leq -c_\tau \| q_1^i -q_2^i\| _2^2$  for all  $(q_{1}^{i},q_{2}^{i})$  , where  $c_{\tau} = \mu_{\mathrm{min}}\ell_{\tau}$  See Lemma B.1 for the definition of  $\mu_{\mathrm{min}}$
Proof of Lemma D.9. (1) For any  $(q_1^i, q_2^i)$  and  $(s_0, a_0^i, a_0^{-i}, s_1)$ , we have
$$
\begin{array}{l} \left\| F ^ {i} \left(q _ {1} ^ {i}, s _ {0}, a _ {0} ^ {i}, a _ {0} ^ {- i}, s _ {1}\right) - F ^ {i} \left(q _ {2} ^ {i}, s _ {0}, a _ {0} ^ {i}, a _ {0} ^ {- i}, s _ {1}\right) \right\| _ {2} ^ {2} \\ = \sum_ {(s, a ^ {i})} ([ F ^ {i} (q _ {1} ^ {i}, s _ {0}, a _ {0} ^ {i}, a _ {0} ^ {- i}, s _ {1}) ] (s, a ^ {i}) - [ F ^ {i} (q _ {2} ^ {i}, s _ {0}, a _ {0} ^ {i}, a _ {0} ^ {- i}, s _ {1}) ] (s, a ^ {i})) ^ {2} \\ = \left(q _ {1} ^ {i} (s _ {0}, a _ {0} ^ {i}) - q _ {2} ^ {i} (s _ {0}, a _ {0} ^ {i})\right) ^ {2} \\ \leq \left\| q _ {1} ^ {i} - q _ {2} ^ {i} \right\| _ {2} ^ {2}. \\ \end{array}
$$
(2) For any  $(s_0, a_0^i, a_0^{-i}, s_1)$ , we have
$$
\begin{array}{l} \| F ^ {i} \left(0, s _ {0}, a _ {0} ^ {i}, a _ {0} ^ {- i}, s _ {1}\right) \| _ {2} ^ {2} = \sum_ {\left(s, a ^ {i}\right)} \left(\left[ F ^ {i} \left(0, s _ {0}, a _ {0} ^ {i}, a _ {0} ^ {- i}, s _ {1}\right) \right] \left(s, a ^ {i}\right)\right) ^ {2} \\ = \left(R _ {i} (s _ {0}, a _ {0} ^ {i}, a _ {0} ^ {- i}) + \gamma v ^ {i} (s _ {1})\right) ^ {2} \\ \leq \frac {1}{(1 - \gamma) ^ {2}}, \\ \end{array}
$$
where the last line follows from  $\| v^i\|_{\infty}\leq 1 / (1 - \gamma)$  and  $|R_{i}(s_{0},a_{0}^{i},a_{0}^{-i})|\leq 1$
(3) We first write down the explicitly the operator  $\overline{F}_k^i (\cdot)$ . Using the definition of  $\mathcal{T}^i (\cdot)$ , we have
$$
\bar {F} _ {k} ^ {i} \left(q ^ {i}\right) (s) = \mu_ {k} (s) \operatorname {d i a g} \left(\pi_ {k} ^ {i} (s)\right) \left(\mathcal {T} ^ {i} \left(v ^ {i}\right) (s) \pi_ {k} ^ {- i} (s) - q ^ {i} (s)\right), \quad \forall s \in \mathcal {S},
$$
Since  $\mu_k(s) \geq \mu_{\min} > 0$  (cf. Lemma B.1 (4)) and  $\mathrm{diag}(\pi_k^i(s))$  has strictly positive diagonal entries (cf. Lemma D.1) for all  $s \in S$  and  $k \geq 0$ , the equation  $\bar{F}_k^i(q^i) = 0$  has a unique solution  $\bar{q}_k^i \in \mathbb{R}^{|\mathcal{S}| |\mathcal{A}^i|}$ , which is given as
$$
\bar {q} _ {k} ^ {i} (s) = \mathcal {T} ^ {i} (v ^ {i}) (s) \pi_ {k} ^ {- i} (s), \quad \forall s \in \mathcal {S}.
$$
(4) Using the expression of  $\bar{F}_k^i (\cdot)$ , we have for any  $q_1^i,q_2^i\in \mathbb{R}^{|\mathcal{S}||\mathcal{A}^i|}$  that
$$
\begin{array}{l} \left(q _ {1} ^ {i} - q _ {2} ^ {i}\right) ^ {\top} \left(\bar {F} _ {k} ^ {i} \left(q _ {1} ^ {i}\right) - \bar {F} _ {k} ^ {i} \left(q _ {2} ^ {i}\right)\right) = - \sum_ {s, a ^ {i}} \mu_ {k} (s) \pi_ {k} ^ {i} \left(a ^ {i} | s\right) \left(q _ {1} ^ {i} \left(s, a ^ {i}\right) - q _ {2} ^ {i} \left(s, a ^ {i}\right)\right) ^ {2} \\ \leq - \min  _ {s, a ^ {i}} \mu_ {k} (s) \pi_ {k} ^ {i} (a ^ {i} | s) \| q _ {1} ^ {i} - q _ {2} ^ {i} \| _ {2} ^ {2} \\ \leq - \mu_ {\min } \ell_ {\tau} \| q _ {1} ^ {i} - q _ {2} ^ {i} \| _ {2} ^ {2} \quad (\text {L e m m a B . 1 a n d L e m m a D . 1}) \\ = - c _ {\tau} \| q _ {1} ^ {i} - q _ {2} ^ {i} \| _ {2} ^ {2}. \\ \end{array}
$$
Next, we establish a negative drift inequality for  $\mathcal{L}_q(k)$ . Using  $\| \cdot \|_2^2$  as a Lyapunov function, we have by Eq. (41) that
$$
\mathbb {E} [ \| q _ {k + 1} ^ {i} - \bar {q} _ {k + 1} ^ {i} \| _ {2} ^ {2} ] = \mathbb {E} [ \| q _ {k + 1} ^ {i} - q _ {k} ^ {i} + q _ {k} ^ {i} - \bar {q} _ {k} ^ {i} + \bar {q} _ {k + 1} ^ {i} \| _ {2} ^ {2} ]
$$
$$
\begin{array}{l} = \mathbb {E} [ \| q _ {k} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {2} ^ {2} ] + \mathbb {E} [ \| q _ {k + 1} ^ {i} - q _ {k} ^ {i} \| _ {2} ^ {2} ] + \mathbb {E} [ \| \bar {q} _ {k} ^ {i} - \bar {q} _ {k + 1} ^ {i} \| _ {2} ^ {2} ] \\ + 2 \alpha_ {k} \mathbb {E} \left[ \left(q _ {k} ^ {i} - \bar {q} _ {k} ^ {i}\right) ^ {\top} \bar {F} _ {k} ^ {i} \left(q _ {k} ^ {i}\right) \right] \\ + 2 \alpha_ {k} \mathbb {E} \left[ \left(F ^ {i} \left(q _ {k} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) - \bar {F} _ {k} ^ {i} \left(q _ {k} ^ {i}\right)\right) ^ {\top} \left(q _ {k} ^ {i} - \bar {q} _ {k} ^ {i}\right) \right] \\ + 2 \mathbb {E} \left[ \left(\bar {q} _ {k} ^ {i} - \bar {q} _ {k + 1} ^ {i}\right) ^ {\top} \left(q _ {k + 1} ^ {i} - q _ {k} ^ {i}\right) \right] \\ + 2 \mathbb {E} \left[ \left(\bar {q} _ {k} ^ {i} - \bar {q} _ {k} ^ {i}\right) ^ {\top} \left(\bar {q} _ {k} ^ {i} - \bar {q} _ {k + 1} ^ {i}\right) \right] \\ \leq \left(1 - 2 \alpha_ {k} c _ {\tau}\right) \mathbb {E} \left[ \| q _ {k} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {2} ^ {2} \right] + \mathbb {E} \left[ \| q _ {k + 1} ^ {i} - q _ {k} ^ {i} \| _ {2} ^ {2} \right] + \mathbb {E} \left[ \| \bar {q} _ {k} ^ {i} - \bar {q} _ {k + 1} ^ {i} \| _ {2} ^ {2} \right] \\ + 2 \mathbb {E} \left[ \left(\bar {q} _ {k} ^ {i} - \bar {q} _ {k + 1} ^ {i}\right) ^ {\top} \left(q _ {k + 1} ^ {i} - q _ {k} ^ {i}\right) \right] \\ + 2 \mathbb {E} \left[ \left(\bar {q} _ {k} ^ {i} - \bar {q} _ {k} ^ {i}\right) ^ {\top} \left(\bar {q} _ {k} ^ {i} - \bar {q} _ {k + 1} ^ {i}\right) \right] \\ + 2 \alpha_ {k} \mathbb {E} \left[ \left(F ^ {i} \left(q _ {k} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) - \bar {F} _ {k} ^ {i} \left(q _ {k} ^ {i}\right)\right) ^ {\top} \left(q _ {k} ^ {i} - \bar {q} _ {k} ^ {i}\right) \right] \tag {42} \\ \end{array}
$$
where the last line follows from Lemma D.9 (4). The terms  $\mathbb{E}[\| q_{k + 1}^i -q_k^i\| _2^2 ],\mathbb{E}[\| \bar{q}_k^i -\bar{q}_{k + 1}^i\| _2^2 ],$ $\mathbb{E}[(\bar{q}_k^i -\bar{q}_{k + 1}^i)^\top (q_{k + 1}^i -q_k^i)],\mathbb{E}[(q_k^i -\bar{q}_k^i)^\top (\bar{q}_k^i -\bar{q}_{k + 1}^i)]$  on the RHS of Eq. (42) are bounded in the following lemma.
Lemma D.10. The following inequalities hold for all  $k \geq 0$ .
(1)  $\mathbb{E}[\| q_{k + 1}^i -q_k^i\| _2^2 ]\leq \frac{4|S|A_{\max}\alpha_k^2}{(1 - \gamma)^2}.$
(2)  $\mathbb{E}[\| \bar{q}_k^i -\bar{q}_{k + 1}^i\| _2^2 ]\leq \frac{4|S|A_{\max}\beta_k^2}{(1 - \gamma)^2}.$
(3)  $\mathbb{E}[\langle q_{k + 1}^i -q_k^i,\bar{q}_k^i -\bar{q}_{k + 1}^i\rangle ]\leq \frac{4|S|A_{\max}\alpha_k\beta_k}{(1 - \gamma)^2}.$
(4)  $\mathbb{E}[\langle q_k^i -\bar{q}_k^i,\bar{q}_k^i -\bar{q}_{k + 1}^i\rangle ]\leq \frac{17|\mathcal{S}|A_{\max}^2\beta_k}{\tau(1 - \gamma)^2}\mathbb{E}[\| q_k^i -\bar{q}_k^i \| _2^2 ] + \frac{\beta_k}{16}\mathbb{E}[\mathcal{L}_\pi (k)].$
Proof of Lemma D.10. (1) For any  $k \geq 0$ , using Eq. (41) and Lemma D.9 (1), we have
$$
\begin{array}{l} \left\| q _ {k + 1} ^ {i} - q _ {k} ^ {i} \right\| _ {2} ^ {2} = \alpha_ {k} ^ {2} \left\| F ^ {i} \left(q _ {k} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) \right\| _ {2} ^ {2} \\ = \alpha_ {k} ^ {2} \| F ^ {i} \left(q _ {k} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) - F ^ {i} \left(0, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) \\ + F ^ {i} \left(0, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) \| _ {2} ^ {2} \\ \leq \alpha_ {k} ^ {2} \left(\| q _ {k} ^ {i} \| _ {2} + \frac {1}{1 - \gamma}\right) ^ {2} \\ \leq \alpha_ {k} ^ {2} \left(\frac {\sqrt {| S | A _ {\max }}}{1 - \gamma} + \frac {1}{1 - \gamma}\right) ^ {2} \quad (\| q _ {k} ^ {i} \| _ {\infty} \leq \frac {1}{1 - \gamma} \text {b y L e m m a D . 1}) \\ \leq \frac {4 | \mathcal {S} | A _ {\operatorname* {m a x}} \alpha_ {k} ^ {2}}{(1 - \gamma) ^ {2}}. \\ \end{array}
$$
The result follows by taking expectation on both sides of the previous inequality.
(2) For any  $k \geq 0$ , using the definition of  $\bar{q}_k$  in Appendix D.1, we have by Lemma D.9 that
$$
\begin{array}{l} \left\| \bar {q} _ {k} ^ {i} - \bar {q} _ {k + 1} ^ {i} \right\| _ {2} ^ {2} = \sum_ {s} \left\| \mathcal {T} ^ {i} (v ^ {i}) (s) \left(\pi_ {k + 1} ^ {- i} (s) - \pi_ {k} ^ {- i} (s)\right) \right\| _ {2} ^ {2} \\ = \beta_ {k} ^ {2} \sum_ {s} \| \mathcal {T} ^ {i} (v ^ {i}) (s) (\sigma_ {\tau} (q _ {k} ^ {- i} (s)) - \pi_ {k} ^ {- i} (s)) \| _ {2} ^ {2} \\ \leq \beta_ {k} ^ {2} \sum_ {s} (\| \mathcal {T} ^ {i} (v ^ {i}) (s) \sigma_ {\tau} (q _ {k} ^ {- i} (s)) \| _ {2} + \| \mathcal {T} ^ {i} (v ^ {i}) (s) \pi_ {k} ^ {- i} (s) \| _ {2}) ^ {2} \\ \leq \frac {4 | \mathcal {S} | A _ {\max} \beta_ {k} ^ {2}}{(1 - \gamma) ^ {2}}. \\ \end{array}
$$
(3) For any  $k\geq 0$  , we have
$$
\langle q _ {k + 1} ^ {i} - q _ {k} ^ {i}, \bar {q} _ {k} ^ {i} - \bar {q} _ {k + 1} ^ {i} \rangle \leq \| q _ {k + 1} ^ {i} - q _ {k} ^ {i} \| _ {2} \| \bar {q} _ {k} ^ {i} - \bar {q} _ {k + 1} ^ {i} \| _ {2} \leq \frac {4 | \mathcal {S} | A _ {\max} \alpha_ {k} \beta_ {k}}{(1 - \gamma) ^ {2}},
$$
where the last inequality follows from Part (1) and Part (2) of this lemma. The result follows by taking expectation on both sides of the previous inequality.
(4) For any  $k\geq 0$  , we have
$$
\begin{array}{l} \left\langle q _ {k} ^ {i} - \bar {q} _ {k} ^ {i}, \bar {q} _ {k} ^ {i} - \bar {q} _ {k + 1} ^ {i} \right\rangle \\ = \beta_ {k} \sum_ {s} \left\langle q _ {k} ^ {i} (s) - \bar {q} _ {k} ^ {i} (s), \mathcal {T} ^ {i} \left(v ^ {i}\right) (s) \left(\sigma_ {\tau} \left(q _ {k} ^ {- i} (s)\right) - \pi_ {k} ^ {- i} (s)\right) \right\rangle \\ \leq \frac {c _ {1} \beta_ {k} \| q _ {k} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {2} ^ {2}}{2} + \frac {\beta_ {k} \sum_ {s} \| \mathcal {T} ^ {i} (v ^ {i}) (s) \left(\sigma_ {\tau} \left(q _ {k} ^ {- i} (s)\right) - \pi_ {k} ^ {- i} (s)\right) \| _ {2} ^ {2}}{2 c _ {1}}, \tag {43} \\ \end{array}
$$
where  $c_{1} > 0$  is an arbitrary positive real number. We next bound the second term on the RHS of the previous inequality. For any  $s \in S$ , we have
$$
\begin{array}{l} \| \mathcal {T} ^ {i} (v ^ {i}) (s) \left(\sigma_ {\tau} \left(q _ {k} ^ {- i} (s)\right) - \pi_ {k} ^ {- i} (s)\right) \| _ {2} \\ = \| \mathcal {T} ^ {i} (v ^ {i}) (s) (\sigma_ {\tau} (q _ {k} ^ {- i} (s)) - \sigma_ {\tau} (\bar {q} _ {k} ^ {- i} (s)) + \sigma_ {\tau} (\mathcal {T} ^ {- i} (v ^ {- i}) (s) \pi_ {k} ^ {i} (s)) - \pi_ {k} ^ {- i} (s)) \| _ {2} \\ \leq \underbrace {\| \mathcal {T} ^ {i} (v ^ {i}) (s) (\sigma_ {\tau} (q _ {k} ^ {- i} (s)) - \sigma_ {\tau} (\bar {q} _ {k} ^ {- i} (s))) \| _ {2}} _ {B _ {1}} \\ + \underbrace {\| \mathcal {T} ^ {i} (v ^ {i}) (s) (\sigma_ {\tau} (\mathcal {T} ^ {- i} (v ^ {- i}) (s) \pi_ {k} ^ {i} (s)) - \pi_ {k} ^ {- i} (s)) \| _ {2}} _ {B _ {2}}. \\ \end{array}
$$
Since the softmax operator  $\sigma_{\tau}(\cdot)$  is  $\frac{1}{\tau}$ -Lipschitz continuous with respect to  $\| \cdot \|_2$  [96, Proposition 4], we have
$$
\begin{array}{l} B _ {1} \leq \| \mathcal {T} ^ {i} (v ^ {i}) (s) \| _ {2} \| \sigma_ {\tau} \left(q _ {k} ^ {- i} (s)\right) - \sigma_ {\tau} \left(\bar {q} _ {k} ^ {- i} (s)\right) \| _ {2} \\ \leq \frac {A _ {\operatorname* {m a x}}}{\tau (1 - \gamma)} \| q _ {k} ^ {- i} (s) - \bar {q} _ {k} ^ {- i} (s) \| _ {2}. \\ \end{array}
$$
We next analyze the term  $B_{2}$ . Using Lemma D.7 (1) and the quadratic growth property of strongly convex functions, we have
$$
\begin{array}{l} B _ {2} = \left\| \right. \mathcal {T} ^ {i} \left(v ^ {i}\right) (s) \left(\sigma_ {\tau} \left(\mathcal {T} ^ {- i} \left(v ^ {- i}\right)\right) (s) \pi_ {k} ^ {i} (s)\right) - \pi_ {k} ^ {- i} (s)\left. \right)\left. \right\| _ {2} \\ \leq \left\| \mathcal {T} ^ {i} \left(v ^ {i}\right) (s) \right\| _ {2} \left\| \sigma_ {\tau} \left(\mathcal {T} ^ {- i} \left(v ^ {- i}\right) (s) \pi_ {k} ^ {i} (s)\right) - \pi_ {k} ^ {- i} (s) \right\| _ {2} \\ \leq \frac {\sqrt {2} A _ {\mathrm {m a x}}}{\sqrt {\tau} (1 - \gamma)} V _ {v, s} (\pi_ {k} ^ {1} (s), \pi_ {k} ^ {2} (s)) ^ {1 / 2}. \\ \end{array}
$$
Combine the upper bounds we obtained for the terms  $B_{1}$  and  $B_{2}$ , we obtain
$$
\begin{array}{l} \sum_ {s} \| \mathcal {T} ^ {i} (v ^ {i}) (s) (\sigma_ {\tau} (q _ {k} ^ {- i} (s)) - \pi_ {k} ^ {- i} (s)) \| _ {2} ^ {2} \\ \leq \sum_ {s} (B _ {1} + B _ {2}) ^ {2} \\ \leq 2 \sum_ {s} \left(B _ {1} ^ {2} + B _ {2} ^ {2}\right) \\ \leq 2 \sum_ {s} \left(\frac {A _ {\operatorname* {m a x}} ^ {2}}{\tau^ {2} (1 - \gamma) ^ {2}} \| q _ {k} ^ {- i} (s)) - \bar {q} _ {k} ^ {- i} (s) \| _ {2} ^ {2} + \frac {2 A _ {\operatorname* {m a x}} ^ {2}}{\tau (1 - \gamma) ^ {2}} V _ {v, s} (\pi_ {k} ^ {1} (s), \pi_ {k} ^ {2} (s))\right) \\ \leq \frac {2 A _ {\mathrm {m a x}} ^ {2}}{\tau^ {2} (1 - \gamma) ^ {2}} \| q _ {k} ^ {- i} - \bar {q} _ {k} ^ {- i} \| _ {2} ^ {2} + \frac {4 | \mathcal {S} | A _ {\mathrm {m a x}} ^ {2}}{\tau (1 - \gamma) ^ {2}} \mathcal {L} _ {\pi} (k). \\ \end{array}
$$
Coming back to Eq. (43), using the previous inequality, we have
$$
\langle q _ {k} ^ {i} - \bar {q} _ {k} ^ {i}, \bar {q} _ {k} ^ {i} - \bar {q} _ {k + 1} ^ {i} \rangle
$$
$$
\begin{array}{l} \leq \frac {c _ {1} \beta_ {k} \| q _ {k} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {2} ^ {2}}{2} + \frac {\beta_ {k} \sum_ {s} \| \mathcal {T} ^ {i} (v ^ {i}) (s) (\sigma_ {\tau} (q _ {k} ^ {- i} (s)) - \pi_ {k} ^ {- i} (s)) \| _ {2} ^ {2}}{2 c _ {1}} \\ \leq \frac {c _ {1} \beta_ {k} \| q _ {k} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {2} ^ {2}}{2} + \frac {A _ {\operatorname* {m a x}} ^ {2} \beta_ {k}}{c _ {1} \tau^ {2} (1 - \gamma) ^ {2}} \| q _ {k} ^ {- i} - \bar {q} _ {k} ^ {- i} \| _ {2} ^ {2} + \frac {2 | \mathcal {S} | A _ {\operatorname* {m a x}} ^ {2} \beta_ {k}}{c _ {1} \tau (1 - \gamma) ^ {2}} \mathcal {L} _ {\pi} (k). \\ \end{array}
$$
Choosing  $c_{1} = \frac{32|S|A_{\max}^{2}}{\tau(1 - \gamma)^{2}}$  in the previous inequality and then taking expectation on both sides, we obtain
$$
\mathbb {E} \left[ \langle q _ {k} ^ {i} - \bar {q} _ {k} ^ {i}, \bar {q} _ {k} ^ {i} - \bar {q} _ {k + 1} ^ {i} \rangle \right] \leq \frac {1 7 | \mathcal {S} | A _ {\max } ^ {2} \beta_ {k}}{\tau (1 - \gamma) ^ {2}} \mathbb {E} \left[ \| q _ {k} ^ {i} - \bar {q} _ {k} ^ {i} \| _ {2} ^ {2} \right] + \frac {\beta_ {k}}{1 6} \mathbb {E} \left[ \mathcal {L} _ {\pi} (k) \right].
$$
We next consider the last term on the RHS of Eq. (42), which involves the difference between the operator  $F^{i}(q_{k}^{i},S_{k},A_{k}^{i},A_{k}^{-i},S_{k + 1})$  and its expected version  $\bar{F}_k^i (q_k^i)$ , and hence can be viewed as the stochastic error due to sampling. The fact that the Markov chain  $\{(S_k,A_k^i,A_k^{-i},S_{k + 1})\}$  is time-inhomogeneous presents a challenge in our analysis. To overcome this challenge, observe that: (1) the policy (hence the transition probability matrix of the induced Markov chain) is changing slowly compared to the  $q$ -function; see Algorithm 3 Line 3, and (2) the stationary distribution as a function of the policy is Lipschitz (cf. Lemma B.1 (3)). These two observations together enable us to develop a refined conditioning argument to handle the time-inhomogeneous Markovian noise. The result is presented in following. Similar ideas were previous used in [23, 24, 65, 41, 76] for finite-sample analysis of single-agent RL algorithms. Recall that we use  $\alpha_{k_1,k_2} = \sum_{k = k_1}^{k_2}\alpha_k$  to simplify the notation.
Lemma D.11 (Proof in Appendix D.7.2). The following inequality holds for all  $k \geq z_k$ :
$$
\mathbb {E} \left[ \left(F ^ {i} \left(q _ {k} ^ {i}, S _ {k}, A _ {k} ^ {i}, A _ {k} ^ {- i}, S _ {k + 1}\right) - \bar {F} _ {k} ^ {i} \left(q _ {k} ^ {i}\right)\right) ^ {\top} \left(q _ {k} ^ {i} - \bar {q} _ {k} ^ {i}\right) \right] \leq \frac {1 7 z _ {k} \alpha_ {k - z _ {k} , k - 1}}{(1 - \gamma) ^ {2}},
$$
where  $z_{k}$  is the mixing time of the Markov chain  $\{S_n\}_{n\geq 0}$  induced by the joint policy  $\pi_k = (\pi_k^1,\pi_k^2)$  with accuracy  $\beta_{k}$ ; see Eq. (11).
When using constant stepsize, we have  $z_{k}\alpha_{k - z_{k},k - 1} = z_{\beta}^{2}\alpha = \mathcal{O}(\alpha \log^{2}(1 / \beta))$ . Since the two step sizes  $\alpha$  and  $\beta$  differ only by a multiplicative constant  $c_{\alpha ,\beta}$ , we have  $\lim_{\alpha \to 0}z_{\beta}^{2}\alpha = 0$ . Similarly, we also have  $\lim_{k\to \infty}z_k\alpha_{k - z_k,k - 1} = 0$  when using diminishing step sizes.
Using the upper bounds we obtained for all the terms on the RHS of Eq. (42), we have the one-step Lyapunov drift inequality for  $q_{k}^{i}$ . Following the same line of analysis, we also obtain the one-step inequality for  $q_{k}^{-i}$ . Adding up the two Lyapunov drift inequalities, we arrive at the following lemma.
Lemma D.12. The following inequality holds for all  $k \geq z_k$  and  $i \in \{1, 2\}$ :
$$
\mathbb {E} [ \mathcal {L} _ {q} (k + 1) ] \leq \left(1 - \alpha_ {k} c _ {\tau}\right) \mathbb {E} [ \mathcal {L} _ {q} (k) ] + \frac {\beta_ {k}}{4} \mathbb {E} [ \mathcal {L} _ {\pi} (k) ] + \frac {1 0 0 | \mathcal {S} | A _ {\max}}{(1 - \gamma) ^ {2}} z _ {k} \alpha_ {k} \alpha_ {k - z _ {k}, k - 1}.
$$