SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[TWYS20] Yi Tian, Yuanhao Wang, Tiancheng Yu, and Suvrit Sra. Provably efficient online agnostic learning in markov games. arXiv preprint arXiv:2010.15020, 2020.
[VDW78] J Van Der Wal. Discounted markov games: Generalized policy iteration method. Journal of Optimization Theory and Applications, 1978.
[vN28] J v. Neumann. Zur theorie der gesellschaftsspiele. Mathematische annalen, 100(1):295-320, 1928.
[WDCW20] Yuanhao Wang, Kefan Dong, Xiaoyu Chen, and Liwei Wang. Q-learning with ub exploration is sample efficient for infinite-horizon mdp. In International Conference on Learning Representations, 2020.
[WHL17] Chen-Yu Wei, Yi-Te Hong, and Chi-Jen Lu. Online reinforcement learning in stochastic games. In Advances in Neural Information Processing Systems, pages 4987–4997, 2017.
[WLZL21a] Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinite-horizon competitive markov games. In Conference on learning theory, pages 4259–4299. PMLR, 2021.
[WLZL21b] Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In International Conference on Learning Representations (ICLR), 2021.
[XCWY20] Qiaomin Xie, Yudong Chen, Zhaoran Wang, and Zhuoran Yang. Learning zero-sum simultaneous-move markov games using function approximation and correlated equilibrium. In Conference on learning theory, pages 3674-3682. PMLR, 2020.
$\left[\mathrm{XZS}^{+}22\right]$  Wei Xiong, Han Zhong, Chengshuai Shi, Cong Shen, and Tong Zhang. A self-play posterior sampling algorithm for zero-sum markov games. In ICLR 2022 Workshop on Gamification and Multiagent Solutions, 2022.
[YM23] Yuepeng Yang and Cong Ma.  $O(T^{-1})$  convergence of optimistic-follow-the-regularized-leader in two-player zero-sum markov games. In International Conference on Learning Representations, 2023.
[  $\mathrm{ZLW}^{+}22$  ] Runyu Zhang, Qinghua Liu, Huan Wang, Caiming Xiong, Na Li, and Yu Bai. Policy optimization for markov games: Unified framework and faster convergence. In Advances in Neural Information Processing Systems, 2022.
[ZTLD22] Yulai Zhao, Yuandong Tian, Jason Lee, and Simon Du. Provably efficient policy optimization for two-player zero-sum markov games. In International Conference on Artificial Intelligence and Statistics, pages 2736-2761. PMLR, 2022.
A Auxiliary Lemmas 15
A.1 Sequence Properties 15
A.2 Properties Related to Exp3-IX 17
A.3 Markov Games 18
A.4 Online Mirror Descent 18
B Last-Iterate Convergence Rate of Algorithm 1 19
C Improved Last-Iterate Convergence under Expectation 23
D Last-Iterate Convergence Rate of Algorithm 2 24
D.1 On the Assumption of Irreducible Markov Game 24
D.2 Part I. Basic Iteration Properties 25
D.3 Part II. Policy Convergence to the Nash of Regularized Game 25
D.4 Part III. Value Convergence 27
D.5 Part IV. Combining 31
E Convergent Analysis of Algorithm 3 32
E.1 Part I. Basic Iteration Properties 32
E.2 Part II. Value Convergence 32
E.3 Part III. Policy Convergence to the Nash of the Regularized Game 37
E.4 Part IV. Combining 40
F Discussions on Convergence Notions for General Markov Games 41
A Auxiliary Lemmas
A.1 Sequence Properties
Lemma 1. Let  $0 < h < 1$ ,  $0 \leq k \leq 2$ , and let  $t \geq \left(\frac{24}{1 - h} \ln \frac{12}{1 - h}\right)^{\frac{1}{1 - h}}$ . Then
$$
\sum_ {i = 1} ^ {t} \left(i ^ {- k} \prod_ {j = i + 1} ^ {t} (1 - j ^ {- h})\right) \leq 9 \ln (t) t ^ {- k + h}.
$$
$$
s \triangleq \left[ (k + 1) t ^ {h} \ln t \right]
$$
We first show that  $s \leq \frac{t}{2}$ . Suppose not, then we have
$$
(k + 1) t ^ {h} \ln t > \frac {t}{2} - 1 \geq \frac {t}{4} \quad (\text {b e c a u s e} t \geq 1 2 > 4)
$$
and thus  $t^{1 - h} < 4(k + 1)\ln t \leq 12\ln t$ . However, by the condition for  $t$  and Lemma 3, it holds that  $t^{1 - h} \geq 12\ln t$ , which leads to contradiction.
Then the sum can be decomposed as
$$
\begin{array}{l} \sum_ {i = 1} ^ {t - s} i ^ {- k} \prod_ {j = i + 1} ^ {t} (1 - j ^ {- h}) + \sum_ {i = t - s + 1} ^ {t} i ^ {- k} \prod_ {j = i + 1} ^ {t} (1 - j ^ {- h}) \\ \leq t \times (1 - t ^ {- h}) ^ {s} + s (t - s + 1) ^ {- k} \\ \leq t \times \left(e ^ {- t ^ {- h}}\right) ^ {s} + s \times \left(\frac {t}{2}\right) ^ {- k} \\ \leq t \times e ^ {- (k + 1) \ln t} + s \times 2 ^ {k} \times t ^ {- k} \\ \leq t ^ {- k} + \left((k + 1) t ^ {h} \ln t + 1\right) \times 2 ^ {k} \times t ^ {- k} \\ \leq 9 \ln (t) t ^ {- k + h}. \\ \end{array}
$$
Lemma 2. Let  $0 < h < 1$ ,  $0 \leq k \leq 2$ , and let  $t \geq \left(\frac{24}{1 - h} \ln \frac{12}{1 - h}\right)^{\frac{1}{1 - h}}$ . Then
$$
\max  _ {1 \leq i \leq t} \left(i ^ {- k} \prod_ {j = i + 1} ^ {t} (1 - j ^ {- h})\right) \leq 4 t ^ {- k}.
$$
$$
\max  _ {\frac {t}{2} \leq i \leq t} \left(i ^ {- k} \prod_ {j = i + 1} ^ {t} (1 - j ^ {- h})\right) \leq \left(\frac {t}{2}\right) ^ {- k} \leq 2 ^ {2} t ^ {- k} = 4 t ^ {- k}
$$
$$
\begin{array}{l} \max  _ {1 \leq i \leq \frac {t}{2}} \left(i ^ {- k} \prod_ {j = i + 1} ^ {t} (1 - j ^ {- h})\right) \leq \left(1 - t ^ {- h}\right) ^ {\frac {t}{2}} \leq \left(\exp \left(- t ^ {- h}\right)\right) ^ {\frac {t}{2}} = \exp \left(- \frac {1}{2} t ^ {1 - h}\right) \\ \stackrel {(a)} {\leq} \exp \left(- \frac {1}{2} \times 1 2 \ln t\right) = \frac {1}{t ^ {6}} \leq t ^ {- k}. \\ \end{array}
$$
where in (a) we use Lemma 3. Combining the two inequalities finishes the proof.
Lemma 3. Let  $0 < h < 1$  and  $t \geq \left(\frac{24}{1 - h} \ln \frac{12}{1 - h}\right)^{\frac{1}{1 - h}}$ . Then  $t^{1 - h} \geq 12 \ln t$ .
Proof. By the condition, we have
$$
t ^ {1 - h} \geq 2 \times \frac {1 2}{1 - h} \ln \frac {1 2}{1 - h}.
$$
Applying Lemma 12, we get
$$
t ^ {1 - h} \geq \frac {1 2}{1 - h} \ln (t ^ {1 - h}) = 1 2 \ln t.
$$
Lemma 4 (Lemma A.1 of [SSBD14]). Let  $a > 0$ . Then  $x \geq 2a\ln(a) \Rightarrow x \geq a\ln(x)$ .
Lemma 5 (Freedman's Inequality). Let  $\mathcal{F}_0\subset \mathcal{F}_1\subset \dots \subset \mathcal{F}_n$  be a filtration, and  $X_{1},\ldots ,X_{n}$  be real random variables such that  $X_{i}$  is  $\mathcal{F}_i$ -measurable,  $\mathbb{E}[X_i|\mathcal{F}_{i - 1}] = 0$ ,  $|X_{i}|\leq b$ , and  $\sum_{i = 1}^{n}\mathbb{E}[X_i^2 |\mathcal{F}_{i - 1}]\leq V$  for some fixed  $b > 0$  and  $V > 0$ . Then with probability at least  $1 - \delta$ ,
$$
\sum_ {i = 1} ^ {n} X _ {i} \leq 2 \sqrt {V \log (1 / \delta)} + b \log (1 / \delta).
$$
A.2 Properties Related to Exp3-IX
In Lemma 6 and Lemma 7, we assume that  $\mathcal{F}_0\subset \mathcal{F}_1\subset \mathcal{F}_2\subset \dots$  is a filtration, and assume that  $x_{i},\ell_{i}$  are  $\mathcal{F}_{i - 1}$ -measurable, where  $x_{i}\in \Delta_{A},\ell_{i}\in [0,1]^{A}$ . Besides,  $a_{i}\in [A]$  and  $\sigma_{i}$  are  $\mathcal{F}_i$ -measurable with  $\mathbb{E}[a_i = a|\mathcal{F}_{i - 1}] = x_{i,a}$  and  $\mathbb{E}[\sigma_i|\mathcal{F}_{i - 1}] = \ell_i$ . Define  $\widehat{\ell}_{i,a} = \frac{\sigma_{i,a}\mathbf{1}[a_i = a]}{x_{i,a} + \beta_i}$  where  $\beta_{i}$  is non-increasing.
Lemma 6 (Lemma 20 of [BJY20]). Let  $c_{1}, c_{2}, \ldots, c_{t}$  be fixed positive numbers. Then with probability at least  $1 - \delta$ ,
$$
\sum_ {i = 1} ^ {t} c _ {i} \left\langle x _ {i}, \ell_ {i} - \widehat {\ell_ {i}} \right\rangle = \mathcal {O} \left(A \sum_ {i = 1} ^ {t} \beta_ {i} c _ {i} + \sqrt {\ln (A / \delta) \sum_ {i = 1} ^ {t} c _ {i} ^ {2}}\right).
$$
Lemma 7 (Adapted from Lemma 18 of [BJY20]). Let  $c_{1}, c_{2}, \ldots, c_{t}$  be fixed positive numbers. Then for any sequence  $x_{1}^{\star}, \ldots, x_{t}^{\star} \in \Delta_{A}$  such that  $x_{i}^{\star}$  is  $\mathcal{F}_{i-1}$ -measurable, with probability at least  $1 - \delta$ ,
$$
\sum_ {i = 1} ^ {t} c _ {i} \left<   x _ {i} ^ {\star}, \widehat {\ell} _ {i} - \ell_ {i} \right> = \mathcal {O} \left(\max _ {i \leq t} \frac {c _ {i} \ln (1 / \delta)}{\beta_ {t}}\right).
$$
Proof. Lemma 18 of [BJY20] states that for any sequence of coefficients  $w_{1}, w_{2}, \dots, w_{t}$  such that  $w_{i} \in [0, 2\beta_{i}]^{A}$  is  $\mathcal{F}_{i-1}$ -measurable, we have with probability  $1 - \delta$ ,
$$
\sum_ {i = 1} ^ {t} c _ {i} \left\langle w _ {i}, \widehat {\ell_ {i}} - \ell_ {i} \right\rangle \leq \max  _ {i \leq t} c _ {i} \log (1 / \delta).
$$
Since  $x_{i}^{\star} \in \Delta_{A}$  and  $\beta_{i}$  is decreasing, we know  $2\beta_{t} \cdot x_{i}^{\star} \in [0,2\beta_{i}]$ . Thus we can apply Lemma 18 of [BJY20] and get with probability  $1 - \delta$ ,
$$
\sum_ {i = 1} ^ {t} c _ {i} \Bigl \langle x _ {i} ^ {\star}, \widehat {\ell} _ {i} - \ell_ {i} \Bigr \rangle = \sum_ {i = 1} ^ {t} \frac {c _ {i}}{2 \beta_ {t}} \Bigl \langle 2 \beta_ {t} \cdot x _ {i} ^ {\star}, \widehat {\ell} _ {i} - \ell_ {i} \Bigr \rangle \leq \max _ {i \leq t} \frac {c _ {i}}{\beta_ {t}} \log (1 / \delta).
$$
Lemma 8 (Lemma 21 of [BJY20]). Let  $c_{1}, c_{2}, \ldots, c_{t}$  be fixed positive numbers. Then with probability at least  $1 - \delta$ , for all  $x^{\star} \in \Delta_{A}$ ,
$$
\sum_ {i = 1} ^ {t} c _ {i} \left\langle x ^ {\star}, \widehat {\ell_ {i}} - \ell_ {i} \right\rangle = \mathcal {O} \left(\max _ {i \leq t} \frac {c _ {i} \ln (A / \delta)}{\beta_ {t}}\right).
$$
Lemma 9. Let  $(x_{1},y_{1})$  and  $(x_{2},y_{2})$  be equilibria of  $f_{1}(\cdot ,\cdot)$  in the domain  $\mathcal{Z}_1$  and  $f_{2}(\cdot ,\cdot)$  in the domain  $\mathcal{Z}_2$  respectively. Suppose that  $\mathcal{Z}_1\subseteq \mathcal{Z}_2$ , and that  $\sup_{(x,y)\in \mathcal{Z}_1}|f_1(x,y) - f_2(x,y)|\leq \epsilon$ . Then for any  $(x,y)\in \mathcal{Z}_2$ ,
$$
f _ {2} (x _ {1}, y) - f _ {2} (x, y _ {1}) \leq 2 \epsilon + 2 d \sup  _ {(\tilde {x}, \tilde {y}) \in \mathcal {Z} _ {2}} \| \nabla f _ {2} (\tilde {x}, \tilde {y}) \| _ {\infty}
$$
where  $d = \max_{z\in \mathcal{Z}_2}\min_{z'\in \mathcal{Z}_1}\| z - z'\| _1$
Proof. Since  $(x_{1},y_{1})$  is an equilibrium of  $f_{1}$ , we have for any  $(x^{\prime},y^{\prime})\in \mathcal{Z}_{1}$
$$
f _ {1} \left(x _ {1}, y ^ {\prime}\right) - f _ {1} \left(x ^ {\prime}, y _ {1}\right) \leq 0,
$$
$$
f _ {2} \left(x _ {1}, y ^ {\prime}\right) - f _ {2} \left(x ^ {\prime}, y _ {1}\right) \leq 2 \epsilon .
$$
For any  $(x,y)\in \mathcal{Z}_2$ , we can find  $(x^{\prime},y^{\prime})\in \mathcal{Z}_{1}$  such that  $\| (x,y) - (x^{\prime},y^{\prime})\|_{1}\leq d$ . Therefore, for any  $(x,y)\in \mathcal{Z}_2$
$$
\begin{array}{l} f _ {2} (x _ {1}, y) - f _ {2} (x, y _ {1}) \\ \leq f _ {2} \left(x _ {1}, y ^ {\prime}\right) - f _ {2} \left(x ^ {\prime}, y _ {1}\right) + \| x - x ^ {\prime} \| _ {1} \| \nabla_ {\mathbf {x}} f _ {2} (x, y) \| _ {\infty} + \| y - y ^ {\prime} \| _ {1} \| \nabla_ {\mathbf {y}} f _ {2} (x, y) \| _ {\infty} \\ \leq 2 \epsilon + 2 d \sup  _ {(\tilde {x}, \tilde {y}) \in \mathcal {Z} _ {2}} \| \nabla f _ {2} (\tilde {x}, \tilde {y}) \| _ {\infty}. \\ \end{array}
$$
A.3 Markov Games
Lemma 10 ([WLZL21a]). For any policy pair  $x, y$ , the duality gap on a two player zero-sum game can be related to duality gap on individual states:
$$
\max _ {s, x ^ {\prime}, y ^ {\prime}} \left(V _ {x, y ^ {\prime}} ^ {s} - V _ {x ^ {\prime}, y} ^ {s}\right) \leq \frac {2}{1 - \gamma} \max _ {s, x ^ {\prime}, y ^ {\prime}} \left(x ^ {s} Q _ {\star} ^ {s} y ^ {\prime s} - x ^ {\prime s} Q _ {\star} ^ {s} y ^ {s}\right).
$$
A.4 Online Mirror Descent
$$
x ^ {\prime} = \underset {\tilde {x} \in \Omega} {\operatorname {a r g m i n}} \left\{\sum_ {a \in \mathcal {A}} \tilde {x} _ {a} \left(\ell_ {a} + \epsilon_ {a} \ln x _ {a}\right) + \frac {1}{\eta} K L (\tilde {x}, x) \right\}
$$
for some convex set  $\Omega \subseteq \Delta_{\mathcal{A}}$ ,  $\ell \in [0,\infty)^{A}$ , and  $\epsilon \in [0,\frac{1}{\eta}]^{A}$ . Then
$$
(x - u) ^ {\top} (\ell + \epsilon \ln x) \leq \frac {K L (u , x) - K L (u , x ^ {\prime})}{\eta} + \eta \sum_ {a \in \mathcal {A}} x _ {a} (\ell_ {a}) ^ {2} + \eta \sum_ {a \in \mathcal {A}} \epsilon_ {a} ^ {2} \ln^ {2} x _ {a}.
$$
for any  $u\in \Omega$  , where  $\epsilon$  ln  $x$  denotes the vector  $(\epsilon_{a}\ln x_{a})_{a\in \mathcal{A}}$
Proof. By the standard analysis of online mirror descent, we have for any  $u \in \Omega$
$$
\left(x - u\right) ^ {\top} (\ell + \epsilon \ln x) \leq \frac {\mathrm {K L} (u , x) - \mathrm {K L} (u , x ^ {\prime})}{\eta} + \left(x - x ^ {\prime}\right) ^ {\top} (\ell + \epsilon \ln x) - \frac {1}{\eta} \mathrm {K L} \left(x ^ {\prime}, x\right).
$$
Below, we abuse the notation by defining  $\mathrm{KL}(\tilde{x},x) = \sum_{a}(\tilde{x}_a\ln \frac{\tilde{x}_a}{x_a} -\tilde{x}_a + x_a)$  without restricting  $\tilde{x}$  to be a probability vector. Then following the analysis in the proof of Lemma 1 of [CLW21], we have
$$
\begin{array}{l} \left(x - x ^ {\prime}\right) ^ {\top} (\ell + \epsilon \ln x) - \frac {1}{\eta} \mathrm {K L} \left(x ^ {\prime}, x\right) \\ \leq \max  _ {y \in \mathbb {R} _ {+} ^ {A}} \left\{\left(x - y\right) ^ {\top} (\ell + \epsilon \ln x) - \frac {1}{\eta} \mathrm {K L} (y, x) \right\} \\ = \frac {1}{\eta} \sum_ {a} x _ {a} \left(\eta \left(\ell_ {a} + \epsilon_ {a} \ln x _ {a}\right) - 1 + e ^ {- \eta \left(\ell_ {a} + \epsilon_ {a} \ln x _ {a}\right)}\right) \\ \leq \frac {1}{\eta} \sum_ {a} x _ {a} \left(\eta \epsilon_ {a} \ln x _ {a} + \eta^ {2} \ell_ {a} ^ {2} - e ^ {- \eta \ell_ {a}} + e ^ {- \eta \ell_ {a}} x _ {a} ^ {- \eta \epsilon_ {a}}\right) \quad (z - 1 \leq z ^ {2} - e ^ {- z} \text {f o r} z \geq 0) \\ = \eta \sum_ {a} x _ {a} \ell_ {a} ^ {2} + \frac {1}{\eta} \sum_ {a} \left(\eta \epsilon_ {a} x _ {a} \ln x _ {a} + e ^ {- \eta \ell_ {a}} \left(x _ {a} ^ {1 - \eta \epsilon_ {a}} - x _ {a}\right)\right) \\ \leq \eta \sum_ {a} x _ {a} \ell_ {a} ^ {2} + \frac {1}{\eta} \sum_ {a} \left(\eta \epsilon_ {a} x _ {a} \ln x _ {a} + \left(x _ {a} ^ {1 - \eta \epsilon_ {a}} - x _ {a}\right)\right) \quad (\eta \ell_ {a} \geq 0 \text {a n d} x _ {a} ^ {1 - \eta \epsilon_ {a}} - x _ {a} \geq 0) \\ \leq \eta \sum_ {a} x _ {a} \ell_ {a} ^ {2} + \frac {1}{\eta} \sum_ {a} \left(\eta \epsilon_ {a} x _ {a} \ln x _ {a} - \eta \epsilon_ {a} x ^ {1 - \eta \epsilon_ {a}} \ln x _ {a}\right) (by Lemma 12) \\ \leq \eta \sum_ {a} x _ {a} \ell_ {a} ^ {2} + \frac {1}{\eta} \sum_ {a} \left(\eta \epsilon_ {a} \ln x _ {a}\right) ^ {2} x _ {a} ^ {1 - \eta \epsilon_ {a}} (by Lemma 12) \\ \leq \eta \sum_ {a} x _ {a} \ell_ {a} ^ {2} + \frac {1}{\eta} \sum_ {a} \left(\eta \epsilon_ {a} \ln x _ {a}\right) ^ {2}. \quad (\eta \epsilon_ {a} \leq 1 \text {a n d} x _ {a} \in (0, 1)) \\ \end{array}
$$
Lemma 12. For  $x \in (0,1)$  and  $y > 0$ , we have  $x^{1 - y} - x \leq -yx^{1 - y}\ln x$ .
$$
\begin{array}{l} x ^ {1 - y} - x = e ^ {(\ln \frac {1}{x}) (y - 1)} - e ^ {(\ln \frac {1}{x}) (- 1)} \\ = y \left(\ln \frac {1}{x}\right) e ^ {\left(\ln \frac {1}{x}\right) \tilde {y}} \quad (\text {f o r s o m e} \tilde {y} \in [ - 1, y - 1 ]) \\ \leq y \left(\ln \frac {1}{x}\right) e ^ {(\ln \frac {1}{x}) (- 1 + y)} \\ = - y (\ln x) x ^ {1 - y} \\ \end{array}
$$
where the second equality is by the mean value theorem.
B Last-Iterate Convergence Rate of Algorithm 1
Proof of Theorem 1. The proof is divided into three parts. In Part I, we establish a descent inequality for  $\mathrm{KL}(z_t^\star, z_t)$ . In Part II, we give an upper bound  $\mathrm{KL}(z_t^\star, z_t)$  by recursively applying the descent inequality. Finally in Part III, we show last-iterate convergence rate on the duality gap of  $z_t = (x_t, y_t)$ . In the proof, we assume without loss of generality that  $t \geq t_0 = \left(\frac{24}{1 - k_\eta - k_\epsilon} \ln \left(\frac{12}{1 - k_\eta - k_\epsilon}\right)\right)^{\frac{1}{1 - k_\eta - k_\epsilon}} = (96 \ln (48))^4$  since the theorem holds trivially for constant  $t$ .
$$
\begin{array}{l} f _ {t} \left(x _ {t}, y _ {t}\right) - f _ {t} \left(x _ {t} ^ {\star}, y _ {t}\right) \\ = \left(x _ {t} - x _ {t} ^ {\star}\right) ^ {\top} G y _ {t} + \epsilon_ {t} \left(\sum_ {a} x _ {t, a} \ln x _ {t, a} - \sum_ {a} x _ {t, a} ^ {\star} \ln x _ {t, a} ^ {\star}\right) \\ = \left(x _ {t} - x _ {t} ^ {\star}\right) ^ {\top} G y _ {t} + \epsilon_ {t} \left(\sum_ {a} \left(x _ {t, a} - x _ {t, a} ^ {\star}\right) \ln x _ {t, a}\right) - \epsilon_ {t} \underbrace {\sum_ {a} x _ {t , a} ^ {\star} \left(\ln x _ {t , a} ^ {\star} - \ln x _ {t , a}\right)} _ {= \mathrm {K L} \left(x _ {t} ^ {\star}, x _ {t}\right)} \\ = (x_{t} - x_{t}^{\star})^{\top}g_{t} - \epsilon_{t}\mathrm{KL}(x_{t}^{\star},x_{t}) + \underbrace{\sum_{a}x_{t,a}\left((Gy_{t})_{a} - \frac{\mathbf{1}[a_{t} = a]\sigma_{t}}{x_{t,a} + \beta_{t}}\right)}_{\triangleq \underline{\xi}_{t}} + \underbrace{\sum_{a}x_{t,a}^{\star}\left(\frac{\mathbf{1}[a_{t} = a]\sigma_{t}}{x_{t,a} + \beta_{t}} - (Gy_{t})_{a}\right)}_{\triangleq \underline{\zeta}_{t}}\\ (\text{by the definition of} g_{t}) \\ \leq \frac {\operatorname {K L} \left(x _ {t} ^ {\star} , x _ {t}\right) - \operatorname {K L} \left(x _ {t} ^ {\star} , x _ {t + 1}\right)}{\eta_ {t}} + \eta_ {t} \sum_ {a} x _ {t, a} \left(\frac {\mathbf {1} [ a _ {t} = a ]}{x _ {t , a} + \beta_ {t}}\right) ^ {2} + \eta_ {t} \sum_ {a} \epsilon_ {t} ^ {2} \ln^ {2} \left(x _ {t, a}\right) - \epsilon_ {t} \operatorname {K L} \left(x _ {t} ^ {\star}, x _ {t}\right) + \underline {{\xi}} _ {t} + \underline {{\zeta}} _ {t} (byLemma11) \\ \leq \frac {(1 - \eta_ {t} \epsilon_ {t}) \mathrm {K L} (x _ {t} ^ {\star} , x _ {t}) - \mathrm {K L} (x _ {t} ^ {\star} , x _ {t + 1})}{\eta_ {t}} + 2 \eta_ {t} \sum_ {a} \left(\frac {\mathbf {1} [ a _ {t} = a ]}{x _ {t , a} + \beta_ {t}} + \epsilon_ {t} ^ {2} \ln^ {2} (x _ {t, a})\right) + \underline {{\xi}} _ {t} + \underline {{\zeta}} _ {t} \\ \leq \frac {\left(1 - \eta_ {t} \epsilon_ {t}\right) \mathrm {K L} \left(x _ {t} ^ {\star} , x _ {t}\right) - \mathrm {K L} \left(x _ {t} ^ {\star} , x _ {t + 1}\right)}{\eta_ {t}} \\ +2\eta_{t}A\times \underbrace{\frac{1}{A}\sum_{a}\left(\frac{\mathbf{1}[a_{t} = a]}{x_{t,a} + \beta_{t}} - 1\right)}_{\triangleq \underline{\lambda}_{t}} +2\eta_{t}A + 2\eta_{t}A\epsilon_{t}^{2}\ln^{2}\left(At^{2}\right) + \underline{\xi}_{t} + \underline{\zeta}_{t} \\ \leq \frac {\left(1 - \eta_ {t} \epsilon_ {t}\right) \mathrm {K L} \left(x _ {t} ^ {\star} , x _ {t}\right) - \mathrm {K L} \left(x _ {t} ^ {\star} , x _ {t + 1}\right)}{\eta_ {t}} + 1 0 \eta_ {t} A \ln^ {2} (A t) + 2 \eta_ {t} A \underline {{\lambda}} _ {t} + \underline {{\xi}} _ {t} + \underline {{\zeta}} _ {t}. (5) \\ \end{array}
$$
Rearranging the above inequality, we get
$$
\begin{array}{l} \mathrm {K L} \left(x _ {t + 1} ^ {\star}, x _ {t + 1}\right) \\ \leq (1 - \eta_ {t} \epsilon_ {t}) \mathrm {K L} (x _ {t} ^ {\star}, x _ {t}) + \eta_ {t} (f _ {t} (x _ {t} ^ {\star}, y _ {t}) - f _ {t} (x _ {t}, y _ {t})) + 1 0 \eta_ {t} ^ {2} A \ln^ {2} (A t) + 2 \eta_ {t} ^ {2} A \underline {{\lambda}} _ {t} + \eta_ {t} \underline {{\xi}} _ {t} + \eta_ {t} \underline {{\zeta}} _ {t} + \underline {{v}} _ {t}, \\ \end{array}
$$
where  $\underline{v}_t \triangleq \mathrm{KL}(x_{t+1}^\star, x_{t+1}) - \mathrm{KL}(x_t^\star, x_{t+1})$ . Similarly, since the algorithm for the  $y$ -player is symmetric, we have the following:
KL  $(y_{t + 1}^{\star},y_{t + 1})$
$$
\leq \left(1 - \eta_ {t} \epsilon_ {t}\right) \mathrm {K L} \left(y _ {t} ^ {\star}, y _ {t}\right) + \eta_ {t} \left(f _ {t} \left(x _ {t}, y _ {t}\right) - f _ {t} \left(x _ {t}, y _ {t} ^ {\star}\right)\right) + 1 0 \eta_ {t} ^ {2} A \ln^ {2} (A t) + 2 \eta_ {t} ^ {2} A \bar {\lambda} _ {t} + \eta_ {t} \bar {\xi} _ {t} + \eta_ {t} \bar {\zeta} _ {t} + \bar {v} _ {t}
$$
$$
\bar {\lambda} _ {t} \triangleq \frac {1}{A} \sum_ {b} \left(\frac {\mathbf {1} [ b _ {t} = b ]}{y _ {t , b} + \beta_ {t}} - 1\right)
$$
$$
\bar {\xi} _ {t} \triangleq \sum_ {b} y _ {t, b} \left(\left(- (G ^ {\top} x _ {t}) _ {b} + 1\right) - \frac {\mathbf {1} [ b _ {t} = b ] (- \sigma_ {t} + 1)}{y _ {t , b} + \beta_ {t}}\right)
$$
$$
\bar {\zeta} _ {t} \triangleq \sum_ {b} y _ {t, b} ^ {\star} \left(\frac {\mathbf {1} [ b _ {t} = b ] (- \sigma_ {t} + 1)}{y _ {t , b} + \beta_ {t}} - \left(- (G ^ {\top} x _ {t}) _ {b} + 1\right)\right)
$$
$$
\bar {v} _ {t} \triangleq \mathrm {K L} \left(y _ {t + 1} ^ {\star}, y _ {t + 1}\right) - \mathrm {K L} \left(y _ {t} ^ {\star}, y _ {t + 1}\right).
$$
Adding the two inequalities above up and using the fact that  $f_{t}(x_{t}^{\star},y_{t}) - f_{t}(x_{t},y_{t}^{\star})\leq 0$  , we get
$$
\mathrm {K L} \left(z _ {t + 1} ^ {\star}, z _ {t + 1}\right) \leq \left(1 - \eta_ {t} \epsilon_ {t}\right) \mathrm {K L} \left(z _ {t} ^ {\star}, z _ {t}\right) + 2 0 \eta_ {t} ^ {2} A \ln^ {2} (A t) + 2 \eta_ {t} ^ {2} A \lambda_ {t} + \eta_ {t} \xi_ {t} + \eta_ {t} \zeta_ {t} + v _ {t}, \tag {6}
$$
where  $\square \triangleq \square +\overline{\square}$  for  $\square = \lambda_t,\xi_t,\zeta_t,v_t$
Part II. Expanding the recursion in Eq. (6), and using the fact that  $1 - \eta_{1}\epsilon_{1} = 0$ , we get
$$
\mathrm {K L} (z _ {t + 1} ^ {\star}, z _ {t + 1}) \leq \underbrace {2 0 A \ln^ {2} (A t) \sum_ {i = 1} ^ {t} w _ {t} ^ {i} \eta_ {i} ^ {2}} _ {\mathrm {t e r m _ {1}}} + \underbrace {2 A \sum_ {i = 1} ^ {t} w _ {t} ^ {i} \eta_ {i} ^ {2} \lambda_ {i}} _ {\mathrm {t e r m _ {2}}} + \underbrace {\sum_ {i = 1} ^ {t} w _ {t} ^ {i} \eta_ {i} \xi_ {i}} _ {\mathrm {t e r m _ {3}}} + \underbrace {\sum_ {i = 1} ^ {t} w _ {t} ^ {i} \eta_ {i} \zeta_ {i}} _ {\mathrm {t e r m _ {4}}} + \underbrace {\sum_ {i = 1} ^ {t} w _ {t} ^ {i} v _ {i}} _ {\mathrm {t e r m _ {5}}}
$$
where  $w_{t}^{i}\triangleq \prod_{j = i + 1}^{t}(1 - \eta_{j}\epsilon_{j})$ . We can bound each term as follows.
By Lemma 1 and the fact that that  $t \geq t_0$ , we have
$$
\mathbf {t e r m} _ {1} \leq \mathcal {O} \left(A \ln^ {2} (A t) \ln (t) t ^ {- 2 k _ {\eta} + (k _ {\eta} + k _ {\epsilon})}\right) = \mathcal {O} \left(A \ln^ {3} (A t) t ^ {- k _ {\eta} + k _ {\epsilon}}\right) = \mathcal {O} \left(A \ln^ {3} (A t) t ^ {- \frac {1}{2}}\right).
$$
Using Lemma 7 with  $x^{\star} = \frac{1}{A}\mathbf{1},\ell_{i} = \mathbf{1}$  for all  $i$ , and  $c_{i} = w_{t}^{i}\eta_{i}^{2}$ , we have with probability  $1 - \frac{\delta}{t^2}$
$$
\mathbf {t e r m} _ {2} = \mathcal {O} \left(\frac {A \ln (A t / \delta) \operatorname* {m a x} _ {i \leq t} c _ {i}}{\beta_ {t}}\right) \stackrel {(a)} {=} \mathcal {O} \left(A \ln (A t / \delta) t ^ {k _ {\beta}} \times t ^ {- 2 k _ {\eta}}\right) = \mathcal {O} \left(A \ln (A t / \delta) t ^ {- \frac {1}{2}}\right)
$$
where in (a) we use Lemma 2 with the fact that  $t \geq t_0$ .
Using Lemma 6 with  $c_{i} = w_{t}^{i}\eta_{i}$ , we have with probability at least  $1 - \frac{\delta}{t^2}$
$$
\begin{array}{l} \mathbf {t e r m} _ {3} \leq \mathcal {O} \left(A \sum_ {i = 1} ^ {t} \beta_ {i} c _ {i} + \sqrt {\ln (A t / \delta) \sum_ {i = 1} ^ {t} c _ {i} ^ {2}}\right) \\ = \mathcal {O} \left(A \sum_ {i = 1} ^ {t} \left[ i ^ {- k _ {\beta} - k _ {\eta}} \prod_ {j = i + 1} ^ {t} \left(1 - j ^ {- k _ {\eta} - k _ {\epsilon}}\right) \right] + \sqrt {\ln (A t / \delta) \sum_ {i = 1} ^ {t} \left[ i ^ {- 2 k _ {\eta}} \prod_ {j = i + 1} ^ {t} \left(1 - j ^ {- k _ {\eta} - k _ {\epsilon}}\right) \right]}\right) \\ = \mathcal {O} \left(A \ln (t) t ^ {- k _ {\beta} + k _ {\epsilon}} + t ^ {- \frac {1}{2} k _ {\eta} + \frac {1}{2} k _ {\epsilon}} \log (A t / \delta)\right) \quad (\text {b y}) \\ = \mathcal {O} \left(A \log (A t / \delta) t ^ {- \frac {1}{4}}\right). \\ \end{array}
$$
Using Lemma 7 with  $c_{i} = w_{t}^{i}\eta_{i}$ , we get with probability at least  $1 - \frac{\delta}{t^2}$
$$
\mathbf {t e r m} _ {4} = \mathcal {O} \left(\frac {\ln (A t / \delta) \operatorname* {m a x} _ {i \leq t} c _ {i}}{\beta_ {t}}\right) \stackrel {(a)} {\leq} \mathcal {O} \left(\ln (A t / \delta) t ^ {- k _ {\eta} + k _ {\beta}}\right) = \mathcal {O} \left(\ln (A t / \delta) t ^ {- \frac {1}{4}}\right)
$$
where  $(a)$  is by Lemma 2 and  $t\geq t_0$
By Lemma 13 and Lemma 1,
$$
\mathbf {t e r m} _ {5} = \mathcal {O} \left(\ln^ {2} (A t) \sum_ {i = 1} ^ {t} w _ {t} ^ {i} t ^ {- 1}\right) = \mathcal {O} \left(\ln^ {3} (A t) t ^ {- 1 + k _ {\eta} + k _ {\epsilon}}\right) = \mathcal {O} \left(\ln^ {3} (A t) t ^ {- \frac {1}{4}}\right).
$$
Combining all terms above, we get that with probability at least  $1 - \frac{3\delta}{t^2}$ ,
$$
\mathrm {K L} \left(z _ {t + 1} ^ {\star}, z _ {t + 1}\right) = \mathcal {O} \left(A \ln^ {3} (A t / \delta) t ^ {- \frac {1}{4}}\right). \tag {7}
$$