SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Corollary 5. For linear bandit with delayed feedback, let  $d$  be the dimension of the feature space. When the arm set  $\mathcal{A}$  is finite, running Algorithm 2 with  $\mathcal{ALG}$  being the Phase Elimination algorithm in Lattimore and Szepesvári (2020), we can upper bound the regret by
$$
\operatorname {R e g r e t} (K) \leq O \left(\frac {1}{q} \sqrt {d K \log A} + \frac {\log^ {2} K}{q} + d _ {\tau} (q) \log K\right)
$$
$$
\operatorname {R e g r e t} (K) \leq O \left(\sqrt {d K \log A} + \mathbb {E} [ \tau ] \log K\right).
$$
When the action set is infinitely large (for example,  $\mathcal{A} = [-1,1]^d$ ), we can use techniques such as covering to obtain regret bounds whose main term scales as  $\tilde{O}(d\sqrt{K})$ . Compared with the result in Howson et al. (2021) which scales as  $\tilde{O}(d\sqrt{K} + d^{3/2}\mathbb{E}[\tau])$ , our bound is tighter in the delay-dependent term as we remove the  $d^{3/2}$  factor.
In tabular MDP settings, using the policy elimination algorithm Zhang et al. (2022b), we can get the following bound.
Corollary 6. For tabular MDP with delayed feedback, running Algorithm 2 with  $\mathcal{ALG}$  being the policy elimination algorithm in Zhang et al. (2022b), we can upper bound the regret by
$$
\operatorname {R e g r e t} (K) \leq O \left(\frac {1}{q} \sqrt {S A H ^ {3} K \iota} + \frac {(H + \log \log K) \log K}{q} + (H + \log \log K) d _ {\tau} (q)\right)
$$
for any  $q \in (0,1)$ , where  $\iota$  is some log factor. In addition, if the delays satisfy Assumption 2, the regret can be upper bounded by
$$
\operatorname {R e g r e t} (K) \leq O \left(\sqrt {S A H ^ {3} K \iota} + H (H + \log \log K) \mathbb {E} [ \tau ]\right).
$$
Compared with the previous result  $\sqrt{H^3SAK\iota} + H^2SA\mathbb{E}[\tau]\log K$  in Howson et al. (2022), our bound removes the  $SA$  factor in the delay-dependent term, therefore is tighter.
We can also apply our framework to linear MDP and MDP with general function approximation settings with delay. These settings have not been studied before, therefore our result is novel. In linear MDP settings, using the algorithm in Gao et al. (2021) we can get the following bound.
Corollary 7. For linear MDP with delayed feedback, let  $d$  be the dimension of the feature space. Running Algorithm 2 with  $\mathcal{ALG}$  being the algorithm in Gao et al. (2021), we can upper bound the regret by
$$
\operatorname {R e g r e t} (K) \leq \tilde {O} \big (\frac {1}{q} \sqrt {d ^ {3} H ^ {4} K} + d H ^ {2} d _ {\tau} (q) \log K \big)
$$
And in MDP with general function approximation, using the algorithm in Kong et al. (2021) we get the following bound.
Corollary 8. For MDP with general function approximation and delayed feedback, let  $\dim_E(\mathcal{F},\epsilon)$  be the  $\epsilon$ -eluder dimension of the function class  $\mathcal{F}$ . Running Algorithm 2 with  $\mathcal{ALG}$  being the algorithm in Kong et al. (2021), we can upper bound the regret by
$$
\operatorname {R e g r e t} (K) \leq \tilde {O} \left(\frac {1}{q} \sqrt {\dim_ {E} ^ {2} (\mathcal {F} , 1 / K) H ^ {4} K} + H ^ {2} \dim_ {E} (\mathcal {F} , 1 / K) d _ {\tau} (q)\right)
$$
$$
\operatorname {R e g r e t} (K) \leq \tilde {O} \left(\sqrt {\dim_ {E} ^ {2} (\mathcal {F} , 1 / K) H ^ {4} K} + H ^ {2} \dim_ {E} (\mathcal {F} , 1 / K) \mathbb {E} [ \tau ]\right).
$$
D Proofs for Tabular Markov Games
D.1 Multi-batched Optimistic Nash-VI
Algorithm 4 Multi-batched Algorithm for Tabular Markov Game
1: Initialize: for any  $(s,a,b,h),\bar{Q}_h(s,a,b)\gets H,\underline{Q}_h(s,a,b)\gets 0,\Delta \gets H,N_h(s,a,b)\gets 0,$  trigger set  $\mathcal{L}\gets \{2^{i - 1}\mid 2^i\leq KH,i = 1,2,\ldots \}$    
2: for episode  $k = 1,2,\dots ,K$  do   
3: if there exists  $(h,s,a,b)$  such that  $N_{h}^{k}(s,a,b)\in \mathcal{L}$  then   
4: for step  $h = H,H - 1,\dots ,1$  do   
5: for  $(s,a,b)\in S\times A\times B$  do   
6:  $t\gets N_h^k (s,a,b)$    
7: if  $t > 0$  then   
8:  $\beta_h^k\gets \mathrm{BONUS}\left(t,\widehat{\nabla}_h\left[\left(\bar{V}_{h + 1} + \underline{V}_{h + 1}\right) / 2\right](s,a,b)\right)$    
9:  $\gamma_h^k\gets (c / H)\widehat{\mathbb{P}}_h(\bar{V}_{h + 1} - \underline{V}_{h + 1})(s,a,b)$    
10:  $\bar{Q}_h^k (s,a,b)\gets \min \left\{\left(r_h + \widehat{\mathbb{P}}_h\bar{V}_{h + 1}\right)(s,a,b) + \gamma_h^k +\beta_h^k,H\right\}$    
11:  $\underline{Q}_h^k (s,a,b)\gets \max \left\{\left(r_h + \widehat{\mathbb{P}}_h\underline{V}_{h + 1}\right)(s,a,b) - \gamma_h^k -\beta_h^k,0\right\}$    
12: end if   
13: end for   
14: for  $s\in S$  do   
15:  $\pi_h^k (\cdot ,\cdot \mid s)\gets \mathrm{CCE}\left(\bar{Q}_h(s,\cdot ,\cdot),\underline{Q}_h(s,\cdot ,\cdot)\right)$    
16:  $\bar{V}_h(s)\gets \left(\mathbb{E}_{\pi_h}\bar{Q}_h\right)(s);\quad \underline{V}_h(s)\gets \left(\mathbb{E}_{\pi_h}\underline{Q}_h\right)(s)$    
17: end for   
18: end for   
19: end if   
20: for step  $h = 1,\dots ,H$  do   
21: take action  $(a_{h},b_{h})\sim \pi_{h}^{k}\left(\cdot ,\cdot \mid s_{h}\right)$  , observe reward  $r_h$  and next state  $s_{h + 1}$    
22: add 1 to  $N_h^k (s_h,a_h,b_h)$  and  $N_{h}^{k}(s_{h},a_{h},b_{h},s_{h + 1})$    
23:  $\widehat{\mathbb{P}}_h(\cdot |s_h,a_h,b_h)\gets N_h^k (s_h,a_h,b_h,\cdot) / N_h^k (s_h,a_h,b_h)$    
24: end for   
25: end for
In this part, we list some of the important lemmas for the tabular Markov game algorithm, most of which are proven in the previous literature Liu et al. (2021). These lemmas are very useful for proving the regret bound in our main theorem.
We denote  $V^{k}, Q^{k}, \pi^{k}, \mu^{k}$  and  $\nu^{k}$  for values and policies at the beginning of the  $k$ -th episode. In particular,  $N_{h}^{k}(s,a,b)$  is the number we have visited the state-action tuple  $(s,a,b)$  at the  $h$ -th step
before the  $k$ -th episode.  $N_h^k (s,a,b,s')$  is defined by the same token. Using this notation, we can further define the empirical transition by  $\widehat{\mathbb{P}}_h^k (s' | s,a,b) \coloneqq N_h^k (s,a,b,s') / N_h^k (s,a,b)$ . If  $N_h^k (s,a,b) = 0$ , we set  $\widehat{\mathbb{P}}_h^k (s' | s,a,b) = 1 / S$ .
We define the empirical variance operator
$$
\widehat {\mathbb {V}} _ {h} ^ {k} V (s, a, b) := \operatorname {V a r} _ {s ^ {\prime} \sim \widehat {\mathbb {P}} _ {h} ^ {k} (\cdot | s, a, b)} V \left(s ^ {\prime}\right) \tag {14}
$$
and true variance operator
$$
\mathbb {V} _ {h} ^ {k} V (s, a, b) := \operatorname {V a r} _ {s ^ {\prime} \sim \mathbb {P} _ {h} ^ {k} (\cdot | s, a, b)} V \left(s ^ {\prime}\right). \tag {15}
$$
The bonus terms can be written as
$$
b _ {h} ^ {k} (s, a, b) = \beta_ {h} ^ {k} (s, a, b) + \gamma_ {h} ^ {k} (s, a, b) \tag {16}
$$
$$
\beta_ {h} ^ {k} (s, a, b) := C \left(\sqrt {\frac {\widehat {\boldsymbol {V}} _ {h} ^ {k} \left[ \left(\bar {V} _ {h + 1} ^ {k} + \underline {{V}} _ {h + 1} ^ {k}\right) / 2 \right] (s , a , b)}{\operatorname* {m a x} \left\{N _ {h} ^ {k} (s , a , b) , 1 \right\}}} + \frac {H ^ {2} S _ {\ell}}{\operatorname* {m a x} \left\{N _ {h} ^ {k} (s , a , b) , 1 \right\}}\right) \tag {17}
$$
and the lower-order term
$$
\gamma_ {h} ^ {k} (s, a, b) := \frac {C}{H} \widehat {\mathbb {P}} _ {h} \left(\bar {V} _ {h + 1} ^ {k} - \underline {{V}} _ {h + 1} ^ {k}\right) (s, a, b) \tag {18}
$$
for some absolute constant  $C > 0$
The following lemma is the standard concentration result.
Lemma 8 (Lemma 21, Liu et al. (2021)). Let  $c_{1}$  be some large absolute constant. Define event  $E_{1}$  to be: for all  $h, s, a, b, s'$  and  $k \in [K]$ ,
$$
\left\{ \begin{array}{l} \left| \left[ \left(\widehat {\mathbb {P}} _ {h} ^ {k} - \mathbb {P} _ {h}\right) V _ {h + 1} ^ {\star} \right] (s, a, b) \right| \leq c _ {1} \left(\sqrt {\frac {\widehat {\mathbb {V}} _ {h} ^ {k} V _ {h + 1} ^ {\star} (s , a , b) \iota}{\max  \left\{N _ {h} ^ {k} (s , a , b) , 1 \right\}}} + \frac {H \iota}{\max  \left\{N _ {h} ^ {k} (s , a , b) , 1 \right\}}\right) \\ \left| \left(\widehat {\mathbb {P}} _ {h} ^ {k} - \mathbb {P} _ {h}\right) (s ^ {\prime} | s, a, b) \right| \leq c _ {1} \left(\sqrt {\frac {\min  \left\{\mathbb {P} _ {h} (s ^ {\prime} | s , a , b) , \widehat {\mathbb {P}} _ {h} ^ {k} (s ^ {\prime} | s , a , b) \right\} \iota}{\max  \left\{N _ {h} ^ {k} (s , a , b) , 1 \right\}}} + \frac {\iota}{\max  \left\{N _ {h} ^ {k} (s , a , b) , 1 \right\}}\right) \\ \left\| \left(\widehat {\mathbb {P}} _ {h} ^ {k} - \mathbb {P} _ {h}\right) (\cdot | s, a, b) \right\| _ {1} \leq c _ {1} \sqrt {\frac {S \iota}{\max  \left\{N _ {h} ^ {k} (s , a , b) , 1 \right\}}}. \end{array} \right. \tag {19}
$$
We have  $\mathbb{P}(E_1) \geq 1 - p$ .
The following lemma shows the upper and lower bounds are indeed the upper and lower bounds of the best responses.
Lemma 9 (Lemma 22, Liu et al. (2021)). Suppose event  $E_{1}$  holds. Then for all  $h, s, a, b$  and  $k \in [K]$ , we have
$$
\left\{ \begin{array}{l} \bar {Q} _ {h} ^ {k} (s, a, b) \geq Q _ {h} ^ {\dagger , \nu^ {k}} (s, a, b) \geq Q _ {h} ^ {\mu^ {k}, \dagger} (s, a, b) \geq \underline {{Q}} _ {h} ^ {k} (s, a, b), \\ \bar {V} _ {h} ^ {k} (s) \geq V _ {h} ^ {\dagger , \nu^ {k}} (s) \geq V _ {h} ^ {\mu^ {k}, \dagger} (s) \geq \underline {{V}} _ {h} ^ {k} (s). \end{array} \right. \tag {20}
$$
Lemma 10 (Lemma 23, Liu et al. (2021)). Suppose event  $E_{1}$  holds. Then for all  $h, s, a, b$  and  $k \in [K]$ , we have
$$
\begin{array}{l} \left| \widehat {\mathbb {V}} _ {h} ^ {k} \left[ \left(\bar {V} _ {h + 1} ^ {k} + \underline {{V}} _ {h + 1} ^ {k}\right) / 2 \right] - \mathbb {V} _ {h} V _ {h + 1} ^ {\pi^ {k}} \right| (s, a, b) \tag {21} \\ \leq 4 H \mathbb {P} _ {h} \left(\bar {V} _ {h + 1} ^ {k} - \underline {{V}} _ {h + 1} ^ {k}\right) (s, a, b) + O \left(1 + \frac {H ^ {4} S \iota}{\max \left\{N _ {h} ^ {k} (s , a , b) , 1 \right\}}\right). \\ \end{array}
$$
Proof of Theorem 4. Suppose event  $E_{1}$  happens. We define
$$
\left\{ \begin{array}{l} \Delta_ {h} ^ {k} := \left(\bar {V} _ {h} ^ {k} - \underline {{V}} _ {h} ^ {k}\right) \left(s _ {h} ^ {k}\right) \\ \zeta_ {h} ^ {k} := \Delta_ {h} ^ {k} - \left(\bar {Q} _ {h} ^ {k} - \underline {{Q}} _ {h} ^ {k}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) \\ \xi_ {h} ^ {k} := \mathbb {P} _ {h} \left(\bar {V} _ {h + 1} ^ {k} - \underline {{V}} _ {h + 1} ^ {k}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) - \Delta_ {h + 1} ^ {k}. \end{array} \right. \tag {22}
$$
Using standard decomposition techniques we have
$$
\begin{array}{l} \Delta_ {h} ^ {k} = \zeta_ {h} ^ {k} + \left(\bar {Q} _ {h} ^ {k} - \underline {{Q}} _ {h} ^ {k}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) \\ \leq \zeta_ {h} ^ {k} + 2 \beta_ {h} ^ {k} + 2 \gamma_ {h} ^ {k} + \widehat {\mathbb {P}} _ {h} ^ {k} \left(\bar {V} _ {h + 1} ^ {k} - \underline {{V}} _ {h + 1} ^ {k}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) \\ \leq \zeta_ {h} ^ {k} + 2 \beta_ {h} ^ {k} + 2 \gamma_ {h} ^ {k} + \mathbb {P} _ {h} \left(\bar {V} _ {h + 1} ^ {k} - \underline {{V}} _ {h + 1} ^ {k}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) + c _ {2} \left(\frac {\mathbb {P} _ {h} \left(\bar {V} _ {h + 1} ^ {k} - \underline {{V}} _ {h + 1} ^ {k}\right) \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right)}{H} + \frac {H ^ {2} S _ {\ell}}{\max  \left\{N _ {h} ^ {k} , 1 \right\}}\right) \\ \leq \zeta_ {h} ^ {k} + 2 \beta_ {h} ^ {k} + \mathbb {P} _ {h} \left(\bar {V} _ {h + 1} ^ {k} - \underline {{V}} _ {h + 1} ^ {k}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) + 2 c _ {2} C \left(\frac {\mathbb {P} _ {h} \left(\bar {V} _ {h + 1} ^ {k} - \underline {{V}} _ {h + 1} ^ {k}\right) \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right)}{H} + \frac {H ^ {2} S _ {\iota}}{\max  \left\{N _ {h} ^ {k} , 1 \right\}}\right) \\ \leq \zeta_ {h} ^ {k} + \left(1 + \frac {c _ {3}}{H}\right) \mathbb {P} _ {h} \left(\bar {V} _ {h + 1} ^ {k} - \underline {{V}} _ {h + 1} ^ {k}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) \\ + 4 c _ {2} C \left(\sqrt {\frac {\iota \widehat {\mathbb {V}} _ {h} ^ {k} \left[ \left(\bar {V} _ {h + 1} ^ {k} + \underline {{V}} _ {h + 1} ^ {k}\right) / 2 \right] \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right)}{\max  \left\{N _ {h} ^ {k} \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right) , 1 \right\}}} + \frac {H ^ {2} S _ {\iota}}{\max  \left\{N _ {h} ^ {k} \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right) , 1 \right\}}\right). \tag {23} \\ \end{array}
$$
By Lemma 10,
$$
\begin{array}{l} \sqrt {\frac {\iota \widehat {\mathbb {V}} _ {h} ^ {k} \left[ \left(\bar {V} _ {h + 1} ^ {k} + \underline {{V}} _ {h + 1} ^ {k}\right) / 2 \right] (s , a , b)}{\max  \left\{N _ {h} ^ {k} (s , a , b) , 1 \right\}}} \\ \leq \left(\sqrt {\frac {\iota \mathbb {V} _ {h} V _ {h + 1} ^ {\pi^ {k}} (s , a , b) + \iota}{\max  \left\{N _ {h} ^ {k} (s , a , b) , 1 \right\}}} + \sqrt {\frac {H \iota \mathbb {P} _ {h} \left(\bar {V} _ {h + 1} ^ {k} - \underline {{V}} _ {h + 1} ^ {k}\right) (s , a , b)}{\max  \left\{N _ {h} ^ {k} (s , a , b) , 1 \right\}}} + \frac {H ^ {2} \sqrt {S _ {\iota}}}{\max  \left\{N _ {h} ^ {k} (s , a , b) , 1 \right\}}\right) \\ \leq c _ {4} \left(\sqrt {\frac {\iota \mathbb {V} _ {h} V _ {h + 1} ^ {\pi^ {k}} (s , a , b) + \iota}{\max  \left\{N _ {h} ^ {k} (s , a , b) , 1 \right\}}} + \frac {\mathbb {P} _ {h} \left(\bar {V} _ {h + 1} ^ {k} - \underline {{V}} _ {h + 1} ^ {k}\right) (s , a , b)}{H} + \frac {H ^ {2} \sqrt {S} \iota}{\max  \left\{N _ {h} ^ {k} (s , a , b) , 1 \right\}}\right), \tag {24} \\ \end{array}
$$
where  $c_{4}$  is some absolute constant. Define  $c_{5} \coloneqq 4c_{2}c_{4}C + c_{3}$  and  $\kappa \coloneqq 1 + c_{5} / H$ . Now we have
$$
\left. \Delta_ {h} ^ {k} \leq \kappa \Delta_ {h + 1} ^ {k} + \kappa \xi_ {h} ^ {k} + \zeta_ {h} ^ {k} + O \left(\sqrt {\frac {\iota \mathbb {V} _ {h} V _ {h + 1} ^ {\pi^ {k}} \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right)}{N _ {h} ^ {k} \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right)}} + \sqrt {\frac {\iota}{N _ {h} ^ {k} \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right)}} + \frac {H ^ {2} S _ {\iota}}{N _ {h} ^ {k} \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right)}}\right) \right\}. \tag {25}
$$
Recursing this argument for  $h \in [H]$  and summing over  $k$ ,
$$
\sum_ {k = 1} ^ {K} \Delta_ {1} ^ {k} \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left[ \kappa^ {h - 1} \zeta_ {h} ^ {k} + \kappa^ {h} \xi_ {h} ^ {k} + O \left(\sqrt {\frac {\iota \mathbb {V} _ {h} V _ {h + 1} ^ {\pi^ {k}} \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right)}{\max  \left\{N _ {h} ^ {k} , 1 \right\}}} + \sqrt {\frac {\iota}{\max  \left\{N _ {h} ^ {k} , 1 \right\}}} + \frac {H ^ {2} S _ {\iota}}{\max  \left\{N _ {h} ^ {k} , 1 \right\}}\right) \right].
$$
It is easy to check that  $\{\zeta_h^k\}$  and  $\{\xi_h^k\}$  are martingales. By Azuma-Hoeffding inequality, with probability at least  $1 - p$
$$
\left\{ \begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \kappa^ {h - 1} \zeta_ {h} ^ {k} \leq O (H \sqrt {H K \iota}) \\ \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {\bar {H}} \kappa^ {h} \xi_ {h} ^ {k} \leq O (H \sqrt {H K \iota}). \end{array} \right. \tag {27}
$$
Now we deal with the rest terms. Note that under the doubling epoch update framework, despite those episodes in which an update is triggered, the number of visits of  $(s,a)$  between the  $i$ -th update of  $\hat{P}_{s,a}$  and the  $i + 1$ -th update of  $\hat{P}_{s,a}$  do not exceed  $2^{i - 1}$ .
We define  $i_{\max} = \max \left\{i \mid 2^{i - 1} \leq KH\right\} = \left\lfloor \log_2(KH) \right\rfloor + 1$ . To facilitate the analysis, we first derive a general deterministic result. Let  $\mathcal{K}$  be the set of indexes of episodes in which no update is triggered. By the update rule we have  $|\mathcal{K}^C| \leq SA(\log_2(KH) + 1)$ . Let  $h_0(k)$  be the first time an update is triggered in the  $k$ -th episode if there is an update in this episode and otherwise  $H + 1$ . Define  $\mathcal{X} = \{(k,h) \mid k \in \mathcal{K}^C, h_0(k) + 1 \leq h \leq H\}$ .
Let  $w = \{w_h^k \geq 0 \mid 1 \leq h \leq H, 1 \leq k \leq K\}$  be a group of non-negative weights such that  $w_h^k \leq 1$  for any  $(k,h) \in [H] \times [K]$  and  $w_h^k = 0$  for any  $(k,h) \in \mathcal{X}$ . As in Zhang et al. (2021b), We can calculate
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \sqrt {\frac {w _ {h} ^ {k}}{N _ {h} ^ {k} \left(s _ {h} ^ {k} , a _ {h} ^ {k}\right)}}
$$
$$
\leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \sum_ {s, a} \sum_ {i = 3} ^ {i _ {\max}} \mathbb {I} \left[ \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) = (s, a), N _ {h} ^ {k} (s, a) = 2 ^ {i - 1} \right] \sqrt {\frac {w _ {h} ^ {k}}{2 ^ {i - 1}}} + 8 S A \left(\log_ {2} (K H) + 4\right)
$$
$$
= \sum_ {s, a} \sum_ {i = 3} ^ {i _ {\mathrm {m a x}}} \frac {1}{\sqrt {2 ^ {i - 1}}} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {I} \left[ \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) = (s, a), N _ {h} ^ {k} (s, a) = 2 ^ {i - 1} \right] \sqrt {w _ {h} ^ {k}} + 8 S A \left(\log_ {2} (K H) + 4\right)
$$
$$
\leq \sum_ {s, a} \sum_ {i = 3} ^ {i _ {\max }} \sqrt {\frac {\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {I} \left[ \left(s _ {h} ^ {k} , a _ {h} ^ {k}\right) = (s , a) , N _ {h} ^ {k} (s , a) = 2 ^ {i - 1} \right]}{2 ^ {i - 1}}}.
$$
$$
\sqrt {\left(\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {I} \left[ \left(s _ {h} ^ {k} , a _ {h} ^ {k}\right) = (s , a) , N _ {h} ^ {k} (s , a) = 2 ^ {i - 1} \right] w _ {h} ^ {k}\right)} + 8 S A (\log_ {2} (K H) + 4)
$$
$$
\leq \sqrt {S A i _ {\max } \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} w _ {h} ^ {k}} + 8 S A \left(\log_ {2} (K H) + 4\right). \tag {28}
$$
By plugging  $w_h^k = \mathbb{V}_h V_{h+1}^{\pi^k} \left( s_h^k, a_h^k, b_h^k \right) \mathbb{I}[(k,h) \notin \mathcal{X}]$  and  $w_h^k = \mathbb{I}[(k,h) \notin \mathcal{X}]$  we have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left[ O \left(\sqrt {\frac {\iota \mathbb {V} _ {h} V _ {h + 1} ^ {\pi^ {k}} \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right)}{\max  \left\{N _ {h} ^ {k} , 1 \right\}}} + \sqrt {\frac {\iota}{\max  \left\{N _ {h} ^ {k} , 1 \right\}}} + \frac {H ^ {2} S _ {\iota}}{\max  \left\{N _ {h} ^ {k} , 1 \right\}}\right) \right] \\ \leq O \left(\sqrt {S A i _ {\max} \iota \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {V} _ {h} V _ {h + 1} ^ {\pi^ {k}} \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right) \mathbb {I} [ (k , h) \notin \mathcal {X} ]} + \sqrt {S A i _ {\max} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {I} [ (k , h) \notin \mathcal {X} ] \iota} + S ^ {2} A \iota \log_ {2} (K H)\right) + | K \\ \leq O \left(\sqrt {S A i _ {\max } \iota \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {V} _ {h} V _ {h + 1} ^ {\pi^ {k}} \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right) \mathbb {I} [ (k , h) \notin \mathcal {X} ]} + \sqrt {S A i _ {\max } K \iota} + S ^ {2} A \iota \log_ {2} (K H)\right) + \left| \mathcal {K} ^ {C} \right| \tag {29} \\ \end{array}
$$
By the Law of total variation and standard martingale concentration (see Lemma C.5 in Jin et al. (2018) for a formal proof), with probability at least  $1 - p$ , we have
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {V} _ {h} V _ {h + 1} ^ {\pi^ {k}} \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) \leq O \left(H ^ {2} K + H ^ {3} \iota\right). \tag {30}
$$
By Pigeon-hole argument,
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \frac {1}{\sqrt {\max  \left\{N _ {h} ^ {k} , 1 \right\}}} \leq \sum_ {s, a, b, h: N _ {h} ^ {K} (s, a, b) > 0} \sum_ {n = 1} ^ {N _ {h} ^ {K} (s, a, b)} \frac {1}{\sqrt {n}} + H S A B \leq O (\sqrt {H ^ {2} S A B K} + H S A B) \tag {31}
$$
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \frac {1}{\max  \left\{N _ {h} ^ {k} , 1 \right\}} \leq \sum_ {s, a, b, h: N _ {h} ^ {K} (s, a, b) > 0} \sum_ {n = 1} ^ {N _ {h} ^ {K} (s, a, b)} \frac {1}{n} + H S A B \leq O (H S A B t). \tag {32}
$$
Putting all relations together, we obtain that
$$
\operatorname {R e g r e t} (K) = \sum_ {k = 1} ^ {K} \left(V _ {1} ^ {\dagger}, \nu^ {k} - V _ {1} ^ {\mu^ {k}, \dagger}\right) (s _ {1}) \leq O \left(\sqrt {H ^ {3} S A B K \iota} + H ^ {3} S ^ {2} A B \iota^ {2}\right) \tag {33}
$$
D.2 Multi-batched Algorithms for Linear Markov Games
Algorithm 5 Multi-batched Algorithm for Linear Markov Games
1: Require: regularization parameter  $\lambda$   
2: Initialize:  $\Lambda_h = \Lambda_h^0 = \lambda I_d$   
3: for episode  $k = 1, 2, \dots, K$  do  
4:  $\Lambda_h^k = \sum_{\tau=1}^{k-1} \phi(s_h^\tau, a_h^\tau, b_h^\tau) \phi(s_h^\tau, a_h^\tau, b_h^\tau)^\top + \lambda I_d$   
5: end for  
6: if  $\exists h \in [H]$ ,  $\operatorname{det}(\Lambda_h^k) > \eta \cdot \operatorname{det}(\Lambda_h)$  then  
7:  $\bar{V}_{H+1}^k \gets 0$ ,  $\underline{V}_{H+1}^k \gets 0$   
8: for step  $h = H, H-1, \dots, 1$  do  
9:  $\Lambda_h \gets \Lambda_h^k$   
10:  $\bar{w}_h^k \gets (\Lambda_h^k)^{-1} \sum_{\tau=1}^{k-1} \phi(s_h^\tau, a_h^\tau, b_h^\tau) \cdot [r_h(s_h^\tau, a_h^\tau, b_h^\tau) + \bar{V}_{h+1}^k(s_{h+1}^\tau)]$   
11:  $\underline{w}_h^k \gets (\Lambda_h^k)^{-1} \sum_{\tau=1}^{k-1} \phi(s_h^\tau, a_h^\tau, b_h^\tau) \cdot [r_h(s_h^\tau, a_h^\tau, b_h^\tau) + \underline{V}_{h+1}^k(s_{h+1}^\tau)]$   
12:  $\Gamma_h^k(\cdot, \cdot, \cdot) \gets \beta \sqrt{\phi(\cdot, \cdot, \cdot)^\top (\Lambda_h^k)^{-1} \phi(\cdot, \cdot, \cdot)}$   
13:  $\bar{Q}_h^k(\cdot, \cdot, \cdot) \gets \Pi_H\{(\bar{w}_h^k)^\top \phi(\cdot, \cdot, \cdot) + \Gamma_h^k(\cdot, \cdot, \cdot)\}$   
14:  $\underline{Q}_h^k(\cdot, \cdot, \cdot) \gets \Pi_H\{(\underline{w}_h^k)^\top \phi(\cdot, \cdot, \cdot) - \Gamma_h^k(\cdot, \cdot, \cdot)\}$   
15: For each  $s$ , let  $\pi_h^k(x) \gets \text{FIND\_CCE}(\bar{Q}_h^k, Q_h^k, s)$  (cf. Algorithm 2 in Xie et al. (2020))  
16:  $\bar{V}_h^k(s) \gets \mathbb{E}_{(a,b) \sim \pi_h^k(s)}\bar{Q}_h^k(s,a,b)$  for each  $s$   
17:  $\underline{V}_h^k(s) \gets \mathbb{E}_{(a,b) \sim \pi_h^k(s)}\underline{Q}_h^k(s,a,b)$  for each  $s$   
18: end for  
19: else  
20:  $\bar{Q}_h^k \gets \bar{Q}_h^{k-1}, Q_h^k \gets Q_h^{k-1}, \pi_h^k \gets \pi_h^{k-1}, \forall h \in [H]$   
21: end if
Proof of Theorem 3. Let  $\{k_1, \dots, k_{N_{\mathrm{batch}}}\}$  be the episodes where the learner updates the policies. Moreover, we define  $b_k = \max \{k_i : i \in [N_{\mathrm{batch}}], k_i \leq k\}$ . Moreover, we denote the marginals of  $\pi_h^k$  as  $\mu_h^k$  and  $\nu_h^k$ .
Part 1: Upper bound of the batch number. By the updating rule, we know there exists one  $h \in [H]$  such that  $\operatorname{det}(\Lambda_h^{k_i}) > \eta \cdot \operatorname{det}(\Lambda_h^{k_{i-1}})$ , which further implies that
$$
\prod_ {h = 1} ^ {H} \det  \left(\Lambda_ {h} ^ {k _ {i}}\right) > \eta \cdot \prod_ {h = 1} ^ {H} \det  \left(\Lambda_ {h} ^ {k _ {i - 1}}\right).
$$
This yields that
$$
\prod_ {h = 1} ^ {H} \det \left(\Lambda_ {h} ^ {k _ {N _ {\text {b a t c h}}}}\right) > \eta^ {N _ {\text {b a t c h}}} \cdot \prod_ {h = 1} ^ {H} \det \left(\Lambda_ {h} ^ {0}\right) = \eta^ {N _ {\text {b a t c h}}} \cdot \lambda^ {d H}. \tag {34}
$$
On the other hand, for any  $h \in [H]$ , we have
$$
\prod_ {h = 1} ^ {H} \det \left(\Lambda_ {h} ^ {k _ {N _ {\text {b a t c h}}}}\right) \leq \prod_ {h = 1} ^ {H} \det \left(\Lambda_ {h} ^ {K + 1}\right). \tag {35}
$$
Furthermore, by the fact that  $\| \phi (\cdot ,\cdot)\| _2\leq 1$  , we have
$$
\Lambda_ {h} ^ {K + 1} = \sum_ {k = 1} ^ {K} \phi \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) \phi \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) ^ {\top} + \lambda I _ {d} \preceq (K + \lambda) \cdot I _ {d},
$$
$$
\prod_ {h = 1} ^ {H} \det  \left(\Lambda_ {h} ^ {K + 1}\right) \leq (K + \lambda) ^ {d H}. \tag {36}
$$
Combining (34), (35), and (36), we obtain that
$$
N _ {\text {b a t c h}} \leq \frac {d H}{\log \eta} \log \left(1 + \frac {K}{\lambda}\right),
$$
which concludes the first part of the proof.
Part2: Regret. We need the following two lemmas to bound the regret.
Lemma 11. For any  $(s, a, b, k, h)$ , it holds that
$$
\underline {{Q}} _ {h} ^ {k} (s, a, b) - 2 (H - h + 1) \epsilon \leq Q _ {h} ^ {\mu^ {k}, *} (s, a, b) \leq Q _ {h} ^ {* \cdot , \nu^ {k}} (s, a, b) \leq \bar {Q} _ {h} (s, a, b) + 2 (H - h + 1) \epsilon ,
$$