SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Most algorithms with  $N_{\mathrm{switch}}$  low-switching cost are multi-batched algorithms with  $N_{\mathrm{switch}}$  batches. For example, the policy elimination algorithm proposed in Zhang et al. (2022b) is a multi-batched
algorithm with  $2H + \log \log K$  batches. And the algorithm for linear MDP with low global switching cost in Gao et al. (2021) is a multi-batched algorithm with  $O(dH\log K)$  batches. On the other hand, the difference between batches and switching cost is that we do not require the policy to be the same during a batch. Therefore, a multi-batched algorithm with  $N$  batches may not be an algorithm with  $N$  switching cost. For example, the Phase Elimination Algorithm in linear bandit with finite arms setting (Lattimore and Szepesvári, 2020) is a multi-batched algorithm with  $O(\log K)$  batches, but the switching cost is  $O(A\log K)$  which is much larger.
4 A Framework for Sequential Decision Making with Delayed Feedback
In this section, we introduce our framework for sequential decision making problems with stochastic delayed feedback. We first describe our framework, which converts any multi-batched algorithm to an algorithm for MSDM with delayed feedback. Then we provide theoretical guarantees for the performance of such algorithms.
Before formally introducing our framework, we first give an intuitive explanation of why multibatched algorithms work for delayed settings. Recall that Lancewicki et al. (2021) gives a lower bound showing that UCB-type algorithms are suboptimal in multi-armed bandits with stochastic delay (see Theorem 1 in Lancewicki et al. (2021)). The main idea of the lower bound is that UCB algorithm will keep pulling a suboptimal arm until the delayed feedback arrives and the policy is updated. Therefore, if the policy updates too frequently, then the algorithm has to spend more time waiting for the update, and the effect of delay will be greater. This observation motivates us to consider multi-batched algorithms, which have a low frequency of policy updating.
For any sequential decision making problem, assume that we have a multi-batched algorithm  $\mathcal{ALG}$  for the problem without delay. Then in the existence of delay, we can still run  $\mathcal{ALG}$ . The difference is, in each batch of the algorithm, we run some extra steps until we observe enough feedback so that the stopping criteria are satisfied. For example, if the delay  $\tau$  is a constant and the stopping criteria is that the policy  $\pi_{k}$  is executed for a pre-defined number of times  $t_k$ , then we only need to run  $\pi_{k}$  for  $t_k + \tau$  times. After the extra steps are finished, we can collect  $t_k$  delayed observations, which meet the original stopping criteria. Then we can continue to the next batch. Algorithm 2 gives a formal description.
Algorithm 2 Multi-batched Algorithm With Delayed Feedback
1: Require: A multi-batched algorithm for the undelayed environment  $\mathcal{ALG}$   
2: Initialize dataset  $\mathcal{D}^0 = \emptyset$   
3: for batch  $m = 1, \dots, M$  do  
4:  $\mathcal{ALG}$  calculates a policy  $\pi^m$  (or a policy sequence) and a stopping criteria  $SC$   
5:  $\mathcal{D} = \emptyset$   
6: while  $SC(\mathcal{D}) = 0$  do  
7:  $k \gets k + 1$   
8: In episode  $k$ , execute  $\pi^m$   
9: Collect trajectory feedback in this batch that is observed by the end of the episode:  $\{o^t : t + \tau_t = k\}$   
10:  $\mathcal{D} = \mathcal{D} \cup \{o^t : t + \tau_t = k\}$   
11: end while  
12: Update dataset  $\mathcal{D}^m = \mathcal{D}^{m-1} \cup \mathcal{D}$   
13: end for  
14: Output: A policy sequence  $\{\pi^m\}$
For the performance of Algorithm 2, we have the following theorem, which relates the regret of delayed MSDM problems with the regret of multi-batched algorithms in the undelayed environment.
Theorem 1. Assume that in the undelayed environment, we have a multi-batched algorithm with  $N_{b}$  batches, and the regret of the algorithm in  $K$  episodes can be upper bounded by  $\operatorname{Regret}(K)$  with probability at least  $1 - \delta$ . Then in the delayed feedback case, with probability at least  $1 - \delta$ , the regret can be upper bounded by
$$
\operatorname {R e g r e t} (K) \leq \frac {1}{q} \widetilde {\operatorname {R e g r e t}} (K) + \frac {2 H N _ {b} \log (K / \delta)}{q} + H N _ {b} d _ {\tau} (q) \tag {6}
$$
for any  $q \in (0,1)$ . In addition, if the delays satisfy Assumption 2, then with probability at least  $1 - \delta$  the regret can be upper bounded by
$$
\operatorname {R e g r e t} (K) \leq \widetilde {\operatorname {R e g r e t}} (K) + H N _ {b} \left(\mathbb {E} [ \tau ] + C _ {\tau}\right), \tag {7}
$$
where  $C_{\tau} = \min \{\sqrt{2v^{2}\log(3KH / (2\delta))},2b\log (3KH / (2\delta))\}$  is a problem-independent constant.
The proof is in Appendix A. The main idea is simple: in each batch, we can control the number of extra steps when waiting for the original algorithm to collect enough delayed feedback. And the extra regret can be bounded by the product of the number of batches and the number of extra steps per batch. Note that in Eq. (6) we do not need Assumption 2. This makes (6) more general.
Applying Theorem 1 to different settings, we will get various types of results. For example, in multi-armed bandit setting, using the Batched Successive Elimination (BaSE) algorithm in Gao et al. (2019), we can get a  $O\left(\frac{1}{q} A \log(K) + d_{\tau}(q) \log(K)\right)$  upper bound from (6), which recovers the SOTA result in Lancewicki et al. (2021) up to a log  $K$  factor in the delay-dependent term. We leave the details to Appendix C, and list the main results in Table 1.
5 Results for Markov Games
As an application, we study Markov games with delayed feedback using our framework. We first develop new multi-batched algorithms for Markov games, then combine them with our framework to obtain regret bounds for the delayed setting.
5.1 Tabular Zero-Sum Markov Game
First we study the tabular zero-sum Markov game setting. The algorithm is formally described in Algorithm 4 in Appendix D.
Our algorithm can be decomposed into two parts: an estimation step and a policy update step. The estimation step largely follows the method in Liu et al. (2021). For each  $(h,s,a,b)$ , we maintain an optimistic estimation  $\bar{Q}_h^k (s,a,b)$  and a pessimistic estimation  $\underline{Q}_h^k (s,a,b)$  of the underlying  $Q_{h}^{\star}(s,a,b)$ , by value iteration with bonus using the empirical estimate of the transition  $\hat{\mathbb{P}}$ . And we compute the Coarse Correlated Equilibrium (CCE) policy  $\pi$  of the estimated value functions Xie et al. (2020). A CCE always exists, and can be computed efficiently.
In the policy update step, we choose the CCE policy w.r.t. the current estimated value function. Note that we use the doubling trick to control the number of updates: we only update the policy when there exists  $(h,s,a,b)$  such that the visiting count  $N_{h}(s,a,b)$  is doubled. This update scheme ensures that the batch number is small.
For the regret bound of Algorithm 4, we have the following theorem.
Theorem 2. For any  $p \in (0,1]$ , letting  $\iota = \log(SABT/p)$ , then with probability at least  $1 - p$ , the regret of Algorithm 4 satisfies
$$
\operatorname {R e g r e t} (K) \leq O \left(\sqrt {H ^ {3} S A B K \iota} + H ^ {3} S ^ {2} A B \iota^ {2}\right).
$$
and the batch number is at most  $O(HSAB\log K)$ .
The proof of Theorem 4 is similar to Theorem 4 in Liu et al. (2021), but the main difference is we have to deal with the sum of the Bernstein-type bonus under the doubling framework. To deal with this we borrow ideas from Zhang et al. (2021b). The whole proof is in Appendix D.
Adapting the algorithm to the delayed feedback setting using our framework, we have the following corollary.
Corollary 1. For tabular Markov game with delayed feedback, running Algorithm 2 with  $\mathcal{ALG}$  being Algorithm 4, we can upper bound the regret by
$$
\operatorname {R e g r e t} (K) \leq O \left(\frac {1}{q} \left(\sqrt {H ^ {3} S A B K \iota} + H ^ {3} S ^ {2} A B \iota^ {2}\right) + H ^ {2} S A B d _ {\tau} (q) \log K\right)
$$
for any  $q \in (0,1)$ . In addition, if the delays satisfy Assumption 2, the regret can be upper bounded by
$$
\operatorname {R e g r e t} (K) \leq O \left(\sqrt {H ^ {3} S A B K \iota} + H ^ {3} S ^ {2} A B \iota^ {2} + H ^ {2} S A B \mathbb {E} [ \tau ] \log K\right).
$$
5.2 Linear Zero-Sum Markov Game
We consider the linear zero-sum Markov game setting, where the reward and transition have a linear structure.
Assumption 3. For each  $(x, a, b) \in S \times \mathcal{A} \times \mathcal{B}$  and  $h \in [H]$ , we have
$$
r _ {h} (x, a, b) = \phi (x, a, b) ^ {\top} \theta_ {h}, \quad \mathbb {P} _ {h} (\cdot \mid x, a, b) = \phi (x, a, b) ^ {\top} \mu_ {h} (\cdot),
$$
where  $\phi : \mathcal{S} \times \mathcal{A} \times \mathcal{A} \to \mathbb{R}^d$  is a known feature map,  $\theta_h \in \mathbb{R}^d$  is an unknown vector and  $\mu_h = (\mu_h^{(i)})_{i \in [d]}$  is a vector of  $d$  unknown (signed) measures on  $\mathcal{S}$ . We assume that  $\| \phi(\cdot, \cdot, \cdot) \| \leq 1$ ,  $\| \theta_h \| \leq \sqrt{d}$  and  $\| \mu_h(\mathcal{S}) \| \leq \sqrt{d}$  for all  $h \in [H]$ , where  $\| \cdot \|$  is the vector  $\ell_2$  norm.
The formal algorithm is in Algorithm 5 in Appendix D.2. It is also divided into two parts. The estimation step is similar to that in Xie et al. (2020), where we calculate an optimistic estimation  $\bar{Q}_h^k(s, a, b)$  and a pessimistic estimation  $\underline{Q}_h^k(s, a, b)$  of the underlying  $Q_h^\star(s, a, b)$  by least square value iteration. And we compute the CCE policy  $\pi$  of the estimated value functions.
In the policy update step, we update the policy to the CCE policy w.r.t. the current estimated value function. Note that instead of updating the policy in every episode, we only update it when there exists  $h \in [H]$  such that  $\operatorname{det}(\Lambda_h^k) > \eta \cdot \operatorname{det}(\Lambda_h)$ . This condition implies that we have collected enough information at one direction. This technique is similar to Gao et al. (2021); Wang et al. (2021).
For the regret bound of Algorithm 5, we have the following theorem.
Theorem 3. In Algorithm 5, let  $\eta = 2$ . Then we have
$$
\operatorname {R e g r e t} (K) \leq \tilde {O} (\sqrt {d ^ {3} H ^ {4} K}).
$$
And the batch number of Algorithm 5 is  $\frac{dH}{\log 2} \log \left(1 + \frac{K}{\lambda}\right)$ .
Proof. See Appendix D.2 for a detailed proof.
Corollary 2. For linear Markov game with delayed feedback, running Algorithm 2 with  $\mathcal{ALG}$  being Algorithm 5, we can upper bound the regret by
$$
\operatorname {R e g r e t} (K) \leq \tilde {O} \left(\frac {1}{q} \sqrt {d ^ {3} H ^ {4} K} + d H ^ {2} d _ {\tau} (q) \log K\right)
$$
$$
\operatorname {R e g r e t} (K) \leq \tilde {O} \left(\sqrt {d ^ {3} H ^ {4} K} + d H ^ {2} \mathbb {E} [ \tau ] \log K\right).
$$
5.3 Tabular General-Sum Markov Game
In this section, we introduce the multi-batched version of V-learning (Jin et al., 2021; Mao and Basar, 2022; Song et al., 2021) for general-sum Markov games. Our goal is to minimize the regret (2). The algorithm is formally described in Algorithm 6 in Appendix D.
The multi-batched V-learning algorithm maintains a value estimator  $V_{h}(s)$ , a counter  $N_{h}(s)$ , and a policy  $\pi_h(\cdot | s)$  for each  $s$  and  $h$ . We also maintain  $S \times H$  different adversarial bandit algorithms. At step  $h$  in episode  $k$ , the algorithm is divided into three steps: policy execution,  $V$ -value update, and policy update. In policy execution step, the algorithm takes action  $a_{h}$  according to  $\pi_{h}^{\tilde{k}}$ , and observes reward  $r_{h}$  and the next state  $s_{h+1}$ , and updates the counter  $N_{h}(s_{h})$ . Note that  $\pi_{h}^{\tilde{k}}$  is updated only when the visiting count of some state  $N_{h}(s)$  doubles. Therefore it ensures a low batch number.
In the  $V$ -value update step, we update the estimated value function by
$$
\tilde {V} _ {h} \left(s _ {h}\right) = \left(1 - \alpha_ {t}\right) \tilde {V} _ {h} \left(s _ {h}\right) + \alpha_ {t} \left(r _ {h} + V _ {h + 1} \left(s _ {h + 1}\right) + \beta (t)\right),
$$
where the learning rate is defined as
$$
\alpha_ {t} = \frac {H + 1}{H + t}, \quad \alpha_ {t} ^ {0} = \prod_ {j = 1} ^ {t} \left(1 - \alpha_ {j}\right), \quad \alpha_ {t} ^ {i} = \alpha_ {i} \prod_ {j = i + 1} ^ {t} \left(1 - \alpha_ {j}\right).
$$
and  $\beta (t)$  is the bonus to promote optimism.
In the policy update step, the algorithm feeds the action  $a_h$  and its "loss"  $\frac{H - r_h + V_{h+1}(s_{h+1})}{H}$  to the  $(s_h, h)^{\text{th}}$  adversarial bandit algorithm. Then it receives the updated policy  $\pi_h(\cdot \mid s_h)$ .
For two-player general sum Markov games, we let the two players run Algorithm 6 independently. Each player uses her own set of bonus that depends on the number of her actions. If we choose the adversarial bandit algorithm as Follow The Regularized Leader (FTRL) algorithm (Lattimore and Szechesváři, 2020), we have the following theorem for the regret of multi-batched V-learning.
Theorem 4. Suppose we choose the adversarial bandit algorithm as FTRL. For any  $\delta \in (0,1)$  and  $K \in \mathbb{N}$ , let  $\iota = \log(HSAK / \delta)$ . Choose learning rate  $\alpha_{t}$  and bonus  $\{\beta(t)\}_{t=1}^{K}$  as  $\beta(t) = c \cdot \sqrt{H^3A\iota / t}$  so that  $\sum_{i=1}^{t} \alpha_{t}^{i} \beta(i) = \Theta(\sqrt{H^3A\iota / t})$  for any  $t \in [K]$ , where  $A = \max_{i} A_{i}$  Then, with probability at least  $1 - \delta$ , after running Algorithm 6 for  $K$  episodes, we have
$$
\operatorname {R e g r e t} (K) \leq O \left(\sqrt {H ^ {5} S A \iota}\right).
$$
And the batch number is  $O(nHS\log K)$ , where  $n$  is the number of players.
Proof. See Appendix D for details.
Finally, adapting the algorithm to the delayed feedback setting using our framework, we have the following corollary.
Corollary 3. For tabular general-sum Markov game with delayed feedback, running Algorithm 2 with  $\mathcal{ALG}$  being Algorithm 6, we can upper bound the regret by
$$
\operatorname {R e g r e t} (K) \leq O \left(\frac {1}{q} \sqrt {H ^ {5} S A \iota} + H ^ {2} n S d _ {\tau} (q) \log K\right)
$$
$$
\operatorname {R e g r e t} (K) \leq O \left(\sqrt {H ^ {5} S A \iota} + H ^ {2} n S \mathbb {E} [ \tau ] \log K\right).
$$
Proof. This corollary is directly implied by Theorems 1 and 4.
In this paper we study delayed feedback in general multi-agent sequential decision making. We propose a novel reduction-based framework, which turns any multi-batched algorithm for sequential decision making with instantaneous feedback into a sample-efficient algorithm that can handle stochastic delays in sequential decision making. Based on this framework, we obtain a complete set of results for (multi-agent) sequential decision making with delayed feedback.
Our work has also shed light on future works. First, it remains unclear whether the result is tight for MDP and Markov game with delayed feedback. Second, it is possible that we can derive a multi-batched algorithm for tabular Markov game with a smaller batch number, since in tabular MDP, Zhang et al. (2022b) gives an algorithm with  $O(H + \log_2\log_2K)$  batch number which is independent of  $S$ . It is not clear whether we can achieve similar results in MDP for Markov games. We believe a tighter result is possible in this setting with a carefully designed multi-batched algorithm.
Liwei Wang is supported by National Key R&D Program of China (2022ZD0114900) and National Science Foundation of China (NSFC62276005). Simon S. Du is supported by NSF IIS 2110170, NSF DMS 2134106, NSF CCF 2212261, NSF IIS 2143493, NSF CCF 2019844, NSF IIS 2229881.
Zhong, H., Xiong, W., Tan, J., Wang, L., Zhang, T., Wang, Z., and Yang, Z. (2022). Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets. arXiv preprint arXiv:2202.07511.
Zhong, H., Yang, Z., Wang, Z., and Jordan, M. I. (2021). Can reinforcement learning find stackelberg-nash equilibria in general-sum markov games with myopic followers? arXiv preprint arXiv:2112.13521.
Zimmert, J. and Seldin, Y. (2020). An optimal algorithm for adversarial bandits with arbitrary delays. In International Conference on Artificial Intelligence and Statistics, pages 3285-3294. PMLR.
We first prove (7). First, since the delays are i.i.d. subexponential random variables, by standard concentration inequality and uniform bound Wainwright (2019), we have the following lemma:
Lemma 5. Let  $\{\tau_t\}_{t=1}^{\infty}$  be i.i.d  $(v, b)$  subexponential random variables with expectation  $\mathbb{E}[\tau]$ . Then we have
$$
\mathbb {P} \left(\exists t \geq 1: \tau_ {t} \geq \mathbb {E} [ \tau ] + C _ {\tau}\right) \leq 1 - \delta , \tag {8}
$$
where  $C_{\tau} = \min \left\{\sqrt{2v^{2}\log\left(\frac{3t}{2\delta}\right)}, 2b\log \left(\frac{3t}{2\delta}\right)\right\}$
Define the good event  $G = \{\exists t \geq 1 : \tau_t \geq \mathbb{E}[\tau] + C_\tau\}$ . By Lemma 5, the good event happens with probability at least  $1 - \delta$ . Therefore, conditioned on the good event happens, in each batch of the algorithm, we only need to run  $\mathbb{E}[\tau] + C_\tau$  episodes and the algorithm can observe enough feedback to finish this batch.
The regret in a batch can be divided into two parts: the first part is the regret caused by running the algorithm in the undelayed environment; the second part is caused by running extra episodes to wait for delayed feedback. The first part can be simply bounded by  $\widetilde{\operatorname{Regret}}(K)$ . For the second part, since each episode contributes at most  $H$  regret, the total regret can be bounded by
$$
\operatorname {R e g r e t} (K) \leq \widetilde {\operatorname {R e g r e t}} (K) + H N _ {b} \left(\mathbb {E} [ \tau ] + C _ {\tau}\right). \tag {9}
$$
For the second inequality, we have the following lemma which reveals the relation between the number of observed feedback and the actual number of episodes using quantile function.
Lemma 6. Denote the number of observed feedback at time  $t$  as  $n_t$ . For any quantile  $q$  and any  $t$ , we have
$$
\Pr \left[ n _ {t + d _ {\tau} (q)} <   \frac {q}{2} t \right] \leq \exp \left(- \frac {q}{8} t\right). \tag {10}
$$
Proof. Define  $\mathbb{I}\left\{\tau_s \leq d_\tau(q)\right\}$  to be an indicator that on time  $s$  that the delay is smaller than  $d_\tau(q)$ . Note that  $\mathbb{E}\left[\mathbb{I}\left\{\tau_s \leq d_\tau(q)\right\} \right] \geq q$ . Thus,
$$
\begin{array}{l} \Pr \left[ n _ {t + d _ {\tau} (q)} <   \frac {q}{2} t \right] \leq \Pr \left[ \sum_ {s = 1} ^ {t} \mathbb {I} \left\{\tau_ {s} \leq d _ {\tau} (q) \right\} <   \frac {q}{2} t \right] \\ \leq \Pr \left[ \sum_ {s = 1} ^ {t} \mathbb {I} \left\{\tau_ {s} \leq d _ {\tau} (q) \right\} <   \frac {1}{2} \sum_ {s = 1} ^ {t} \mathbb {E} \left[ \mathbb {I} \left\{\tau_ {s} \leq d _ {\tau} (q) \right\} \right] \right] \tag {11} \\ \leq \exp \left(- \frac {1}{8} \sum_ {s = 1} ^ {t} \mathbb {E} \left[ \mathbb {I} \left\{\tau_ {s} \leq d _ {\tau} (q) \right\} \right]\right) \leq \exp \left(- \frac {q}{8} t\right), \\ \end{array}
$$
where the third inequality follows from the relative Chernoff bound, and the last inequality is since
$$
\sum_ {s = 1} ^ {t} \mathbb {E} \left[ \mathbb {I} \left\{\tau_ {s} \leq d _ {\tau} (q) \right\} \right] \geq q \cdot t.
$$
Lemma 6 means, if the length of the batch in the undelayed environment  $m > \frac{\log(K / \delta)}{q}$ , then with probability at least  $1 - \frac{\delta}{K}$ , if we run the batch for  $\frac{m}{q} + d_{\tau}(q)$  episodes, we will observe at least  $m$  feedbacks, which is enough for the algorithm to finish this batch. On the other hand, if  $m \leq \frac{\log(K / \delta)}{q}$ , then the regret of this batch can be trivially bounded by  $\frac{H \log(K / \delta)}{q}$ . Hence the total regret can be bounded by
$$
\operatorname {R e g r e t} (K) \leq \operatorname {R e g r e t} \left(\frac {K}{q} + d _ {\tau} (q) N _ {b}\right) \leq \frac {1}{q} \widetilde {\operatorname {R e g r e t}} (K) + \frac {2 H N _ {b} \log (K / \delta)}{q} + H N _ {b} d _ {\tau} (q), \tag {12}
$$
which finishes the proof.
B Details of Algorithms for Linear Bandits
The phase elimination algorithm for linear bandits in Lattimore and Szepesvári (2020) is as follows. This algorithm is naturally a multi-batched algorithm, with each phase as one batch. Since the length of phase  $\ell$  is  $O(2^{\ell})$ , the number of phases is  $O(\log K)$ . Note that in each phase the policy is pre-calculated at the beginning of the phase, and does not rely on the data of current phase.
Algorithm 3 Phase Elimination
1: Input:  $\mathcal{A} \subset \mathbb{R}^d$ ,  $\beta > 1$ ,  $\delta$
2: Set  $\ell = 1$  and let  $\mathcal{A}_1 = \mathcal{A}$
3: Let  $t_\ell = t$  be the current timestep and find  $G$ -optimal design  $\pi_{\ell} \in \mathcal{P}(\mathcal{A}_{\ell})$  with  $\operatorname{Supp}(\pi_{\ell}) \leq d(d + 1)/2$  that maximises
$$
\log \det  V (\pi_ {\ell}) \text {s u b j e c t} \sum_ {a \in \mathcal {A} _ {\ell}} \pi_ {\ell} (a) = 1
$$
4: Let  $\varepsilon_{\ell} = 2^{-\ell}$  and
$$
T _ {\ell} (a) = \left\lceil \frac {2 d \pi_ {\ell} (a)}{\varepsilon_ {\ell} ^ {2}} \log \left(\frac {k \ell (\ell + 1)}{\delta}\right)\right\rceil \text {a n d} T _ {\ell} = \sum_ {a \in \mathcal {A} _ {\ell}} T _ {\ell} (a)
$$
5: Choose each action  $a \in \mathcal{A}_{\ell}$  exactly  $T_{\ell}(a)$  times.
6: Calculate the empirical estimate:
$$
\hat {\theta} _ {\ell} = V _ {\ell} ^ {- 1} \sum_ {t = t _ {\ell}} ^ {t _ {\ell} + T _ {\ell}} A _ {t} X _ {t} \quad \text {w i t h} \quad V _ {\ell} = \sum_ {a \in \mathcal {A} _ {\ell}} T _ {\ell} (a) a a ^ {\top}
$$
using observable data in this phase.
7: Eliminating low reward arms:
$$
\mathcal {A} _ {\ell + 1} = \left\{a \in \mathcal {A} _ {\ell}: \max  _ {b \in \mathcal {A} _ {\ell}} \left\langle \hat {\theta} _ {\ell}, b - a \right\rangle \leq 2 \varepsilon_ {\ell} \right\}.
$$
8:  $\ell \gets \ell + 1$  and go to Line 2.
Theorem 7 (Theorem 22.1 in Lattimore and Szepesvári (2020)). With probability at least  $1 - \delta$ , the regret of Algorithm 5 satisfies
$$
\operatorname {R e g r e t} (K) \leq C \sqrt {K d \log \left(\frac {A \log (K)}{\delta}\right)}, \tag {13}
$$
where  $C > 0$  is a universal constant. If  $\delta = O(1 / K)$ , then  $\mathbb{E}\left[\mathrm{Regret}(K)\right] \leq C\sqrt{Kd\log(AK)}$  for an appropriately chosen universal constant  $C > 0$ .
For infinite action set  $\mathcal{A} = [-1,1]^d$ , we use a covering technique: we find a  $\epsilon$ -covering of the action space, and run Phase Elimination on the covering set. The covering number of the set  $[-1,1]^d$  is  $O(d\log(1/\epsilon))$ . And the error caused by using  $\epsilon$ -covering will lead to a  $O(\epsilon K)$  increase in regret. Therefore, by setting  $\epsilon = \frac{1}{K}$ , the regret of the whole algorithm is  $O(d\sqrt{K}\log K)$ .
C Results for Bandits and MDPs
We now apply Theorem 1 to different bandit and MDP problems.
In multi-armed bandit settings, using the BaSE algorithm Gao et al. (2019), we can get the following bound.
Corollary 4. For multi-armed bandit with delayed feedback, running Algorithm 2 with  $\mathcal{ALG}$  being the BaSE algorithm in Gao et al. (2019), we can upper bound the regret by
$$
\operatorname {R e g r e t} (K) \leq O \left(\frac {A \log K}{q \min _ {i \neq \star} \Delta_ {i}} + \frac {\log^ {2} K}{q} + d _ {\tau} (q) \log K\right)
$$
for any  $q \in (0,1)$ , where  $\Delta_i$  is the optimal gap of the  $i^{th}$  arm. In addition, if the delays satisfy Assumption 2, the regret can be upper bounded by
$$
\operatorname {R e g r e t} (K) \leq O \left(\frac {A \log K}{\min  _ {i \neq \star} \Delta_ {i}} + \mathbb {E} [ \tau ] \log K\right).
$$
This result recovers the SOTA result in Lancewicki et al. (2021) up to a log  $K$  factor in the delay-dependent term.
In linear bandit settings, using the Phase Elimination algorithm (Lattimore and Szepesvári, 2020), we can get the following bound.