SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\underline {{V}} _ {h} ^ {k} (s) - 2 (H - h + 2) \epsilon \leq V _ {h} ^ {\mu^ {k}, *} (s) \leq V _ {h} ^ {*, \nu^ {k}} (s) \leq \bar {V} _ {h} ^ {k} (s) + 2 (H - h + 2) \epsilon .
$$
Proof. See Lemma 5 of Xie et al. (2020) for a detailed proof.
Lemma 12. For all  $(s, a, b, h, k)$  and any fixed policy pair  $(\mu, \nu)$ , it holds with probability at least  $1 - \delta$  that
$$
\left| \left\langle \phi (s, a, b), \bar {w} _ {h} ^ {b _ {k}} \right\rangle - Q _ {h} ^ {\mu , \nu} (s, a, b) - \mathbb {P} _ {h} \left(\bar {V} _ {h + 1} ^ {b _ {k}} - V _ {h + 1} ^ {\mu , \nu}\right) (s, a, b) \right| \leq \Gamma_ {h} ^ {b _ {k}} (s, a, b),
$$
$$
\left| \left\langle \phi (s, a, b), \underline {{w}} _ {h} ^ {b _ {k}} \right\rangle - Q _ {h} ^ {\mu , \nu} (s, a, b) - \mathbb {P} _ {h} \left(\underline {{V}} _ {h + 1} ^ {b _ {k}} - V _ {h + 1} ^ {\mu , \nu}\right) (s, a, b) \right| \leq \Gamma_ {h} ^ {b _ {k}} (s, a, b).
$$
Proof. See Lemma 3 of Xie et al. (2020) for a detailed proof.
$$
\delta_ {h} ^ {k} := \bar {V} _ {h} ^ {b _ {k}} \left(s _ {h} ^ {k}\right) - \underline {{V}} _ {h} ^ {b _ {k}} \left(s _ {h} ^ {k}\right),
$$
$$
\zeta_ {h} ^ {k} := \mathbb {E} [ \delta_ {h + 1} ^ {k} \mid s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k} ] - \delta_ {h + 1} ^ {k},
$$
$$
\bar {\gamma} _ {h} ^ {k} := \mathbb {E} _ {(a, b) \sim \pi_ {h} ^ {k} (s _ {h} ^ {k})} [ \bar {Q} _ {h} ^ {b _ {k}} (s _ {h} ^ {k}, a, b) ] - \bar {Q} _ {h} ^ {b _ {k}} (s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}),
$$
$$
\underline {{\gamma}} _ {h} ^ {k} := \mathbb {E} _ {(a, b) \sim \pi_ {h} ^ {k} (s _ {h} ^ {k})} [ \bar {Q} _ {h} ^ {b _ {k}} (s _ {h} ^ {k}, a, b) ] - \bar {Q} _ {h} ^ {b _ {k}} (s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}).
$$
By Lemma 11, we have
$$
\begin{array}{l} V _ {1} ^ {*}, \nu^ {k} (s _ {1}) - V _ {1} ^ {\pi^ {k}, *} \leq \bar {V} _ {1} ^ {k} (s _ {1}) - \underline {{V}} _ {1} ^ {k} (s _ {1}) + 8 K \epsilon \\ \leq \bar {V} _ {1} ^ {k} (s _ {1}) - \underline {{V}} _ {1} ^ {k} (s _ {1}) + \frac {8}{K} \\ = \bar {V} _ {1} ^ {b _ {k}} (s _ {1}) - \underline {{V}} _ {1} ^ {b _ {k}} (s _ {1}) + \frac {8}{K} \\ = \delta_ {1} ^ {k} + \frac {8}{K}, \tag {37} \\ \end{array}
$$
where the second inequality uses that  $\epsilon = 1 / (KH)$ . By definition, we have
$$
\begin{array}{l} \delta_ {h} ^ {k} = \bar {V} _ {h} ^ {b _ {k}} \left(s _ {h} ^ {k}\right) - \bar {V} _ {h} ^ {b _ {k}} \left(s _ {h} ^ {k}\right) \\ = \mathbb {E} _ {(a, b) \sim \pi_ {h} ^ {k} \left(s _ {h} ^ {k}\right)} [ \bar {Q} _ {h} ^ {k} \left(s _ {h} ^ {k}, a, b\right) ] - \mathbb {E} _ {(a, b) \sim \pi_ {h} ^ {k} \left(s _ {h} ^ {k}\right)} [ \underline {{Q}} _ {h} ^ {b _ {k}} \left(s _ {h} ^ {k}, a, b\right) ] \\ = \bar {Q} _ {h} ^ {k} \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) - \underline {{Q}} _ {h} ^ {b _ {k}} \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) + \bar {\gamma} _ {h} ^ {k} - \underline {{\gamma}} _ {h} ^ {k}. \tag {38} \\ \end{array}
$$
Meanwhile, for any  $(s,a,b,k,h)$ , we have
$$
\begin{array}{l} \bar {Q} _ {h} ^ {b _ {k}} (s, a, b) - \underline {{Q}} _ {h} ^ {b _ {k}} (s, a, b) = \left[ \left(\bar {w} _ {h} ^ {b _ {k}}\right) ^ {\top} \phi (s, a, b) + \Gamma_ {h} ^ {b _ {k}} (s, a, b) \right] - \left[ \left(\underline {{w}} _ {h} ^ {b _ {k}}\right) ^ {\top} \phi (s, a, b) - \Gamma_ {h} ^ {b _ {k}} (s, a, b) \right] \\ = \left(\bar {w} _ {h} ^ {b _ {k}} - \underline {{w}} _ {h} ^ {b _ {k}}\right) ^ {\top} \phi (s, a, b) + 2 \Gamma_ {h} ^ {b _ {k}} (s, a, b). \\ \end{array}
$$
By Lemma 12, we further have
$$
\bar {Q} _ {h} ^ {b _ {k}} (s, a, b) - \underline {{Q}} _ {h} ^ {b _ {k}} (s, a, b) \leq \mathbb {P} _ {h} \left(\bar {V} _ {h + 1} ^ {b _ {k}} - \underline {{V}} _ {h + 1} ^ {b _ {k}}\right) (s, a, b) + 4 \Gamma_ {h} ^ {b _ {k}} (s, a, b). \tag {39}
$$
Plugging (39) into (38) gives that
$$
\begin{array}{l} \delta_ {h} ^ {k} \leq \mathbb {P} _ {h} (\bar {V} _ {h + 1} ^ {b _ {k}} - \underline {{V}} _ {h + 1} ^ {b _ {k}}) (s, a, b) + 4 \Gamma_ {h} ^ {b _ {k}} (s, a, b) + \bar {\gamma} _ {h} ^ {k} - \underline {{\gamma}} _ {h} ^ {k} \\ = \delta_ {h + 1} ^ {k} + \zeta_ {h} ^ {k} + \bar {\gamma} _ {h} ^ {k} - \underline {{\gamma}} _ {h} ^ {k} + 4 \Gamma_ {h} ^ {b _ {k}} (s, a, b). \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \delta_ {1} ^ {k} \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\zeta_ {h} ^ {k} + \bar {\gamma} _ {h} ^ {k} - \underline {{\gamma}} _ {h} ^ {k}\right) + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \Gamma_ {h} ^ {b _ {k}} \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) \\ \leq \tilde {O} (H \cdot \sqrt {H K}) + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \Gamma_ {h} ^ {b _ {k}} \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right), \tag {40} \\ \end{array}
$$
where the last inequality uses the Azuma-Hoeffding inequality. Note that
$$
\frac {\Gamma_ {h} ^ {b _ {k}} \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right)}{\Gamma_ {h} ^ {k} \left(s _ {h} ^ {k} , a _ {h} ^ {k} , b _ {h} ^ {k}\right)} \leq \sqrt {\frac {\operatorname* {d e t} \left(\Lambda_ {h} ^ {k}\right)}{\operatorname* {d e t} \left(\Lambda_ {h} ^ {b _ {k}}\right)}} \leq \sqrt {\eta},
$$
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \Gamma_ {h} ^ {b _ {k}} \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) \leq \sqrt {\eta} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \Gamma_ {h} ^ {k} \left(s _ {h} ^ {k}, a _ {h} ^ {k}, b _ {h} ^ {k}\right) \leq \tilde {O} \left(\sqrt {\eta d ^ {3} H ^ {4} K}\right), \tag {41}
$$
where the last inequality uses the elliptical potential lemma (Abbasi-Yadkori et al., 2011). Combining (37), (40), and (41), together with the fact that  $\eta = 2$ , we obtain
$$
\operatorname {R e g r e t} (K) \leq \tilde {O} (\sqrt {d ^ {3} H ^ {4} K}),
$$
D.3 Multi-batched V-learning
In this section, we introduce the multi-batched version of V-learning (Jin et al., 2021) for general-sum Markov games. Our goal is to minimize the following notion of regret. Let  $V_{i,1}^{\pi}(s_1)$  be the expected cumulative reward that the  $i^{\text{th}}$  player will receive if the game starts at initial state  $s_1$  at the  $1^{\text{st}}$  step and all players follow joint policy  $\pi$ . Let  $\hat{\pi}^{k}$  be the policy executed at the  $k^{\text{th}}$  episode. The regret is defined as
$$
\operatorname {R e g r e t} (K) = \sum_ {k = 1} ^ {K} \max  _ {j} \left(V _ {j, h} ^ {\dagger , \hat {\pi} _ {- j, h} ^ {k}} - V _ {j, h} ^ {\hat {\pi} _ {h} ^ {k}}\right) (s _ {1}), \tag {42}
$$
where  $V_{j,h}^{\dagger,\hat{\pi}_{-j,h}^{k}}$  is the best response of the  $j^{\mathrm{th}}$  player when other players follow  $\hat{\pi}$ .
Algorithm 6 Multi-batched V-learning
1: Initialize:  $V_{h}(s) \gets H + 1 - h, N_{h}(s) \gets 0, \pi_{h}(a \mid s) \gets 1 / A$ , trigger set  $\mathcal{L} \gets \{2^{i-1} \mid 2^i \leq KH, i = 1, 2, \ldots\}$   
2: for episode  $k = 1, \ldots, K$  do  
3: Receive  $s_1$ .  
4: for  $h = 1, \ldots, H$  do  
5: take action  $a_{h} \sim \pi_{h}^{k}(\cdot \mid s_{h})$ , observe  $r_{h}$  and  $s_{h+1}$ .  
6:  $t = N_{h}(s_{h}) \gets N_{h}(s_{h}) + 1$ .  
7:  $\tilde{V}_{h}(s_{h}) \gets (1 - \alpha_{t}) \tilde{V}_{h}(s_{h}) + \alpha_{t} (r_{h} + V_{h+1}(s_{h+1}) + \beta(t))$   
8:  $V_{h}(s_{h}) \gets \min \left\{H + 1 - h, \tilde{V}_{h}(s_{h})\right\}$   
9:  $\pi_{h}^{k}(\cdot \mid s_{h}) \gets ADV\_BANDIT\_UPDATE\left(a_{h}, \frac{H - r_{h} - V_{h+1}(s_{h+1})}{H}\right)$  on  $(s_{h}, h)^{\text{th}}$  adversarial bandit.  
10: end for  
11: if  $\exists s, N_{h}(s) \in \mathcal{L}$  then  
12:  $k \gets k$   
13: end if  
14: end for
The multi-batched V-learning algorithm maintains a value estimator  $V_{h}(s)$ , a counter  $N_{h}(s)$ , and a policy  $\pi_h(\cdot \mid s)$  for each  $s$  and  $h$ . We also maintain  $S\times H$  different adversarial bandit algorithms. At step  $h$  in episode  $k$ , the algorithm is divided into three steps: policy execution,  $V$ -value update, and policy update. In policy execution step, the algorithm takes action  $a_{h}$  according to  $\pi_h^{\tilde{k}}$ , and observes reward  $r_h$  and the next state  $s_{h + 1}$ , and updates the counter  $N_{h}(s_{h})$ . Note that  $\pi_h^{\tilde{k}}$  is updated only when the visiting count of some state  $N_{h}(s)$  doubles. Therefore it ensures a low batch number.
In the policy update step, the algorithm feeds the action  $a_h$  and its "loss"  $\frac{H - r_h + V_{h+1}(s_{h+1})}{H}$  to the  $(s_h, h)^{\text{th}}$  adversarial bandit algorithm. Then it receives the updated policy  $\pi_h(\cdot | s_h)$ .
The ADV_BANDIT_UPDATE subroutine Consider a multi-armed bandit problem with adversarial loss, where we denote the action set by  $\mathcal{B}$  with  $|\mathcal{B}| = B$ . At round  $t$ , the learner determines a policy  $\theta_t \in \Delta_{\mathcal{B}}$ , and the adversary chooses a loss vector  $\ell_t \in [0,1]^B$ . Then the learner takes
an action  $b_{t}$  according to policy  $\theta_{t}$ , and receives a noisy bandit feedback  $\tilde{\ell}_{t}(b_{t}) \in [0,1]$ , where  $\mathbb{E}\left[\tilde{\ell}_{t}(b_{t}) \mid \ell_{t}, b_{t}\right] = \ell_{t}(b_{t})$ . Then, the adversarial bandit algorithm performs updates based on  $b_{t}$  and  $\tilde{\ell}_{t}(b_{t})$ , and outputs the policy for the next round  $\theta_{t+1}$ .
We first state our requirement for the adversarial bandit algorithm used in V-learning, which is to have a high probability weighted external regret guarantee as follows.
Assumption 4. For any  $t \in \mathbb{N}$  and any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ , we have
$$
\max  _ {\theta \in \Delta_ {\mathcal {B}}} \sum_ {i = 1} ^ {t} \alpha_ {t} ^ {i} \left[ \left\langle \theta_ {i}, \ell_ {i} \right\rangle - \left\langle \theta , \ell_ {i} \right\rangle \right] \leq \xi (B, t, \log (1 / \delta)). \tag {43}
$$
We further assume the existence of an upper bound  $\Xi(B, t, \log(1/\delta)) \geq \sum_{t'=1}^{t} \xi\left(B, t', \log(1/\delta)\right)$  where (i)  $\xi(B, t, \log(1/\delta))$  is non-decreasing in  $B$  for any  $t, \delta$ ; (ii)  $\Xi(B, t, \log(1/\delta))$  is concave in  $t$  for any  $B, \delta$ .
As is shown in Jin et al. (2021), the Follow-the-Regularized-Leader (FTRL) algorithm satisfies the assumption with bounds  $\xi(B, t, \log(1/\delta)) \leq O\left(\sqrt{HB \log(B/\delta)/t}\right)$  and  $\Xi(B, t, \log(1/\delta)) \leq O\left(\sqrt{HBt \log(B/\delta)}\right)$ . And we will use FTRL as the subroutine in our algorithm.
For two-player general sum Markov games, we let the two players run Algorithm 6 independently. Each player will use her own set of bonus that depends on the number of her actions. We have the following theorem for the regret of multi-batched V-learning.
Theorem 13. Suppose we choose subroutine ADV_BANDIT_UPDATE as FTRL. For any  $\delta \in (0,1)$  and  $K\in \mathbb{N}$ , let  $\iota = \log (HSAK / \delta)$ . Choose learning rate  $\alpha_{t}$  and bonus  $\{\beta (t)\}_{t = 1}^{K}$  as  $\beta (t) = c\cdot \sqrt{H^3A\iota / t}$  so that  $\sum_{i = 1}^{t}\alpha_{t}^{i}\beta (i) = \Theta \left(\sqrt{H^{3}A\iota / t}\right)$  for any  $t\in [K]$ , where  $A = \max_{i}A_{i}$  Then, with probability at least  $1 - \delta$ , after running Algorithm for  $K$  episodes, we have
$$
\operatorname {R e g r e t} (K) \leq O \left(\sqrt {H ^ {5} S A \iota}\right). \tag {44}
$$
And the batch number is  $O(HS\log K)$ .
Proof. The proof mainly follows Jin et al. (2021). Below we list some useful lemmas, which have already been proved in Jin et al. (2021).
Lemma 14 (Lemma 10, Jin et al. (2021)). The following properties hold for  $\alpha_{t}^{i}$ :
1.  $\frac{1}{\sqrt{t}} \leq \sum_{i=1}^{t} \frac{\alpha_t^i}{\sqrt{i}} \leq \frac{2}{\sqrt{t}}$  and  $\frac{1}{t} \leq \sum_{i=1}^{t} \frac{\alpha_t^i}{i} \leq \frac{2}{t}$  for every  $t \geq 1$ .
2.  $\max_{i\in [t]}\alpha_t^i\leq \frac{2H}{t}$  and  $\sum_{i = 1}^{t}\left(\alpha_{t}^{i}\right)^{2}\leq \frac{2H}{t}$  for every  $t\geq 1$
3.  $\sum_{t = i}^{\infty}\alpha_t^i = 1 + \frac{1}{H}$  for every  $i\geq 1$
The next lemma shows that  $V_{h}^{k}$  is an optimistic estimation of  $V_{j,h}^{\dagger,\hat{\pi}_{-j,h}^{k}}$
Lemma 15 (Lemma 13, Jin et al. (2021)). For any  $\delta \in (0,1]$ , with probability at least  $1 - \delta$ , for any  $(s,h,k,j) \in S \times [H] \times [K] \times [m]$ ,  $V_{j,h}^{k}(s) \geq V_{h}^{\dagger,\nu^{k}}(s)$
And we define the pessimistic pessimistic V-estimations  $\underline{V}$  that are defined similarly as  $\tilde{V}$  and  $V$ . Formally, let  $t = N_h^k(s)$ , and suppose  $s$  was previously visited at episodes  $k^1, \ldots, k^t < k$  at the  $h$ -th step. Then
$$
\underline {{V}} _ {j, h} ^ {k} (s) = \max  \left\{0, \sum_ {i = 1} ^ {t} \alpha_ {t} ^ {i} \left[ r _ {j, h} \left(s, \boldsymbol {a} _ {h} ^ {k ^ {i}}\right) + \underline {{V}} _ {j, h + 1} ^ {k ^ {i}} \left(s _ {h + 1} ^ {k ^ {i}}\right) - \beta (i) \right] \right\}. \tag {45}
$$
The next lemma shows  $\underline{V}_{j,h}^{k}(s)$  is a lower bound of  $V_{j,h}^{\hat{\pi}_h^k}(s)$ .
Lemma 16 (Lemma 14, Jin et al. (2021)). For any  $\delta \in (0,1]$ , with probability at least  $1 - \delta$ , the following holds for any  $(s,h,k,j) \in \mathcal{S} \times [H] \times [K] \times [m]$  and any player  $j, \underline{V}_{j,h}^{k}(s) \leq V_{j,h}^{\hat{\pi}_{h}^{k}}(s)$ .
It remains to bound the gap  $\sum_{k=1}^{K} \max_j \left( V_{1,j}^k - \underline{V}_{1,j}^k \right) (s_1)$ . For each player  $j$  we define  $\delta_{j,h}^k \coloneqq V_{j,h}^k (s_h^k) - \underline{V}_{j,h}^k (s_h^k) \geq 0$ . The non-negativity here is a simple consequence of the update rule and induction. We need to bound  $\delta_h^k \coloneqq \max_j \delta_{j,h}^k$ . Let  $n_h^k = N_h^k (s_h^k)$  and suppose  $s_h^k$  was previously visited at episodes  $k^1, \ldots, k^{n_h^k} < k$  at the  $h$ -th step. Now by the update rule,
$$
\begin{array}{l} \delta_ {j, h} ^ {k} = V _ {j, h} ^ {k} \left(s _ {h} ^ {k}\right) - \underline {{V}} _ {j, h} ^ {k} \left(s _ {h} ^ {k}\right) \\ \leq \alpha_ {n _ {h} ^ {k}} ^ {0} H + \sum_ {i = 1} ^ {n _ {h} ^ {k}} \alpha_ {n _ {h} ^ {k}} ^ {i} \left[ \left(V _ {j, h + 1} ^ {k ^ {i}} - \underline {{V}} _ {j, h + 1} ^ {k ^ {i}}\right) \left(s _ {h + 1} ^ {k ^ {i}}\right) + 2 \beta_ {j} ^ {\tilde {k}} (i) \right] \\ \leq \alpha_ {n _ {h} ^ {k}} ^ {0} H + \sum_ {i = 1} ^ {n _ {h} ^ {k}} \alpha_ {n _ {h} ^ {k}} ^ {i} \left[ \left(V _ {j, h + 1} ^ {k ^ {i}} - \underline {{V}} _ {j, h + 1} ^ {k ^ {i}}\right) \left(s _ {h + 1} ^ {k ^ {i}}\right) + 4 \beta_ {j} ^ {k} (i) \right] \tag {46} \\ = \alpha_ {n _ {h} ^ {k}} ^ {0} H + \sum_ {i = 1} ^ {n _ {h} ^ {k}} \alpha_ {n _ {h} ^ {k}} ^ {i} \delta_ {j, h + 1} ^ {k ^ {i}} + O \left(H \xi \left(A _ {j}, n _ {h} ^ {k}, \iota\right) + \sqrt {H ^ {3} \iota / n _ {h} ^ {k}}\right) \\ \end{array}
$$
The third line holds because we use FTRL as the subroutine,  $\beta_{j}(t) = c\cdot \sqrt{H^{3}A_{j}\iota / t}$ , and due to our doubling scheme we have  $\beta (n_h^{\tilde{k}})\leq 2\beta (n_h^k)$ . And in the last step we have used  $\sum_{i = 1}^{t}\alpha_{t}^{i}\beta_{j,i} = \Theta \left(H\xi \left(A_{j},t,\iota\right) + \sqrt{H^{3}A\iota / t}\right)$ .
The remaining step is the same as Jin et al. (2021). Summing the first two terms w.r.t.  $k$
$$
\sum_ {k = 1} ^ {K} \alpha_ {n _ {h} ^ {k}} ^ {0} H = \sum_ {k = 1} ^ {K} H \mathbb {I} \left\{n _ {h} ^ {k} = 0 \right\} \leq S H, \tag {47}
$$
$$
\sum_ {k = 1} ^ {K} \sum_ {i = 1} ^ {n _ {h} ^ {k}} \alpha_ {n _ {h} ^ {k}} ^ {i} \delta_ {h + 1} ^ {k ^ {i}} \leq \sum_ {k ^ {\prime} = 1} ^ {K} \delta_ {h + 1} ^ {k ^ {\prime}} \sum_ {i = n _ {h} ^ {k ^ {\prime}} + 1} ^ {\infty} \alpha_ {i} ^ {n _ {h} ^ {k ^ {\prime}}} \leq \left(1 + \frac {1}{H}\right) \sum_ {k = 1} ^ {K} \delta_ {h + 1} ^ {k}. \tag {48}
$$
Putting them together,
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \delta_ {h} ^ {k} = \sum_ {k = 1} ^ {K} \alpha_ {n _ {h} ^ {k}} ^ {0} H + \sum_ {k = 1} ^ {K} \sum_ {i = 1} ^ {n _ {h} ^ {k}} \alpha_ {n _ {h} ^ {k}} ^ {i} \delta_ {h + 1} ^ {k ^ {i}} + \sum_ {k = 1} ^ {K} O \left(H \xi \left(A, n _ {h} ^ {k}, \iota\right) + \sqrt {\frac {H ^ {3} \iota}{n _ {h} ^ {k}}}\right) \tag {49} \\ \leq H S + \left(1 + \frac {1}{H}\right) \sum_ {k = 1} ^ {K} \delta_ {h + 1} ^ {k} + \sum_ {k = 1} ^ {K} O \left(H \xi \left(A, n _ {h} ^ {k}, \iota\right) + \sqrt {\frac {H ^ {3} \iota}{n _ {h} ^ {k}}}\right). \\ \end{array}
$$
Recursing this argument for  $h \in [H]$  gives
$$
\sum_ {k = 1} ^ {K} \delta_ {1} ^ {k} \leq e S H ^ {2} + e \sum_ {h = 1} ^ {H} \sum_ {k = 1} ^ {K} O \left(H \xi \left(A, n _ {h} ^ {k}, \iota\right) + \sqrt {\frac {H ^ {3} \iota}{n _ {h} ^ {k}}}\right). \tag {50}
$$
By pigeonhole argument,
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \left(H \xi \left(A, n _ {h} ^ {k}, \iota\right) + \sqrt {H ^ {3} \iota / n _ {h} ^ {k}}\right) = O (1) \sum_ {s} \sum_ {n = 1} ^ {N _ {h} ^ {K} (s)} \left(H \xi (A, n, \iota) + \sqrt {\frac {H ^ {3} \iota}{n}}\right) \\ \leq O (1) \sum_ {s} \left(H \Xi \left(A, N _ {h} ^ {K} (s), \iota\right) + \sqrt {H ^ {3} N _ {h} ^ {K} (s) \iota}\right) \tag {51} \\ \leq O \left(H S \Xi (A, K / S, \iota) + \sqrt {H ^ {3} S K \iota}\right), \\ \end{array}
$$
where in the last step we have used concavity. Finally take the sum w.r.t  $h \in [H]$ , and plugging  $\Xi(B, t, \log(1/\delta)) \leq O(\sqrt{HBt\log(B/\delta)})$  we have
$$
\operatorname {R e g r e t} (K) \leq O \left(\sqrt {H ^ {5} S A \iota}\right). \tag {52}
$$