SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yunchang Yang $^{1*}$  Han Zhong $^{1*}$  Tianhao Wu $^{2*}$  Bin Liu $^{3}$  Liwei Wang $^{1,4}$  Simon S. Du $^{5}$
<sup>1</sup>Center for Data Science, Peking University
$^{2}$ University of California, Berkeley  $^{3}$ Zhejiang Lab
$^{4}$ National Key Laboratory of General Artificial Intelligence,
$^{5}$ University of Washington
We study stochastic delayed feedback in general sequential decision-making problems, which include bandits, single-agent Markov decision processes (MDPs), and Markov games (MGs). We propose a novel reduction-based framework, which turns any multi-batched algorithm for sequential decision making with instantaneous feedback into a sample-efficient algorithm that can handle stochastic delays in sequential decision-making problems. By plugging different multi-batched algorithms into our framework, we provide several examples demonstrating that our framework not only matches or improves existing results for bandits, tabular MDPs, and tabular MGs, but also provides the first line of studies on delays in sequential decision making with function approximation. In summary, we provide a complete set of sharp results for single-agent and multi-agent sequential decision-making problems with delayed feedback.
Delayed feedback is a common and fundamental problem in sequential decision making (Sutton and Barto, 2018; Lattimore and Szepesvári, 2020; Zhang et al., 2021a). Taking the recommendation system as an example, delayed feedback is an inherent part of this problem. Specifically, the learner adjusts her recommendation after receiving users' feedback, which may take a random delay after the recommendation is issued. More examples include but are not limited to robotics (Mahmood et al., 2018) and video steaming (Changuel et al., 2012). Furthermore, the delayed feedback issue is more serious in multi-agent systems (Chen et al., 2020) since observing the actions of other agents' may happen with a varying delay.
Although handling delayed feedback is a prominent challenge in practice, the theoretical understanding of delays in sequential decision making is limited. Even assuming the delays are stochastic (delays are sampled from a fixed distribution), the nearly minimax optimal result is only obtained by works on multi-armed bandits and contextual bandits (Agarwal and Duchi, 2011; Dudik et al., 2011; Joulani et al., 2013; Vernade et al., 2020, 2017; Gael et al., 2020). For more complex problems such as linear bandits (Lancewicki et al., 2021) and tabular reinforcement learning (RL) (Howson et al., 2022), the results are suboptimal. Meanwhile, to the best of our knowledge, there are no theoretical guarantees in RL with function approximation and multi-agent RL settings. Hence, we focus on stochastic delays and aim to answer the following two questions in this work:
1. Can we provide sharper regret bounds for basic models such as linear bandits and tabular RL?
*Equal contribution. Correspondence to Yunchang Yang (yangyc@oku.edu.cn) and Han Zhong (hanzhong@stu.pku.edu.cn).
Table 1: Comparison of bounds for MSDM with delayed feedback. We denote by  $K$  the number of rounds the agent plays (episodes in RL),  $A$  and  $B$  the number of actions (or arms),  $A_{\max} = \max_i A_i$ ,  $S$  the number of states,  $H$  the length of an episode,  $d$  the linear feature dimension, and  $\dim_E(\mathcal{F}, 1 / K)$  the eluder dimension of general function class  $\mathcal{F}$ ,  $d_{\tau}(q)$  is the quantile function of the random variable  $\tau$  and  $n$  is the number of players.
Setting	Previous Result	This Work
Multi-armed bandit	O(A log(K)/q mini≠, Δi+dτ(q) log K)(Lancewicki et al., 2021)	O(A log K/q mini≠, Δi+ log2K/q+dτ(q) log K)
O(A log K/mini≠, Δi+ E[τ] log K)
Linear bandit	O(d√K+d3/2E[τ])(Howson et al., 2021)O(d√K+E[τ])(Vakili et al., 2023), concurrent work	O(1/q d√K+dτ(q))
O(d√K+E[τ])
Tabular MDP	O(√H3SAK+H2SAE[τ] log K)(Howson et al., 2022)	O(1/q √SAH3K+H(H+log log K)dτ(q))
O(√SAH3K+H(H+log log K)E[τ])
Linear MDP	—	O(1/q √d3H4K+dH2dτ(q) log K)
O(√d3H4K+dH2E[τ])
RL with general function approximation	—	O(1/q √dimE2(F,1/K)H4K+H2dimE(F,1/K)dτ(q))
O(√dimE2(F,1/K)H4K+H2dimE(F,1/K)E[τ])
Tabular two-player zero-sum MG	—	O(1/q (√H3SABK+H3S2AB)+H2SABdτ(q) log K)
O(√H3SABK+H3S2AB+H2SABE[τ] log K).
Linear two-player zero-sum MG	—	O(1/q √d3H4K+dH2dτ(q) log K)
O(√d3H4K+dH2E[τ] log K)
Tabular multi-player general-sum MG	O(H3√SAmax+H3√SE[τ]K)(Zhang et al., 2022a)	O(1/q √H5SAmaxK+H2nSdτ(q) log K)
O(√H5SAmaxK+H2nSE[τ] log K)
2. Can we handle delays in (multi-agent) sequential decision making in the context of function approximation?
We answer these two questions affirmatively by proposing a reduction-based framework for stochastic delays in both single-agent and multi-agent sequential decision making. Our contributions are summarized below.
Our Contributions. Our main contribution is proposing a new reduction-based framework for both single-agent and multi-agent sequential decision making with stochastic delayed feedback. The proposed framework can convert any multi-batched algorithm (cf. Section 3) for sequential decision making with instantaneous feedback to a provably efficient algorithm that is capable of handling stochastic delayed feedback in sequential decision making. Unlike previous works which require case by case algorithm design and analysis, we provide a unified framework to solve a wide range of problems all together.
Based on this framework, we obtain a complete set of results for single-agent and multi-agent sequential decision making with delayed feedback. In specific, our contributions can be summarized as follows:
- New framework: We propose a generic algorithm that can be integrated with any multi-batched algorithm in a black-box fashion. Meanwhile, we provide a unified theoretical analysis for the proposed generic algorithm.
- Improved results and new results: By applying our framework to different settings, we obtain state-of-the-art regret bounds for multi-armed bandits, and derive sharper results for linear bandits and tabular RL, which significantly improve existing results (Lancewicki et al., 2021; Howson et al., 2022). We also provide the first result for single-agent RL with delayed feedback in the context of linear or even general function approximation;
- New algorithms: We show that delayed feedback in Markov games (MGs) can also be handled by our framework. By plugging several new proposed multi-batched algorithms for MGs into our framework, we not only improve recent results on tabular MGs (Zhang et al., 2022a), but
also present a new result for linear MGs. As a byproduct, we provide the first line of study on multi-batched algorithms for MGs, which might be of independent interest.
Our results and comparisons with existing works are summarized in Table 1.
Bandit/MDP with delayed feedback. Stochastically delayed feedback has been studied a lot in the multi-armed bandit and contextual bandit problems (Agarwal and Duchi, 2011; Dudik et al., 2011; Joulani et al., 2013; Vernade et al., 2020, 2017; Gael et al., 2020; Huang et al., 2023). A recent work (Lancewicki et al., 2021) shows that under stochastic delay, elimination algorithm performs better than UCB algorithm, and gives tight upper and lower bounds for multi-armed bandit setting. Howson et al. (2021) solves the linear bandit setting by adapting LinUCB algorithm. The concurrent work of Vakili et al. (2023) provides a bound for the kernel bandit setting with delayed feedback. Our work does not consider the kernel setting, but we conjecture that our framework can handle this problem by designing a multi-batched algorithm for kernel bandits.
In RL, despite practical importance, there is limited literature on stochastic delay in RL. Lancewicki et al. (2022) and Jin et al. (2022b) considered adversarial delays. Their regret depends on the sum of the delays, the number of states and the number of steps per episode. However, when reducing their results to stochastic delay, the bounds will be too loose. Howson et al. (2022) considered stochastic delay in tabular MDP, but their bound is not tight. As far as we know, there is no work considering linear MDP and general function approximation with delayed feedback.
Another line of work considers adversarially delayed feedback (Lancewicki et al., 2022; Jin et al., 2022b; Quanrud and Khashabi, 2015; Thune et al., 2019; Bistritz et al., 2019; Zimmert and Seldin, 2020; Ito et al., 2020; Gyorgy and Joulani, 2021; Van Der Hoeven and Cesa-Bianchi, 2022). In the adversarial setting the state-of-the-art result is of form  $\sqrt{K + D}$ , where  $K$  is the number of the episodes and  $D$  is the total delay. Note that when adapting their result to the stochastic setting, the bound becomes  $\sqrt{K + K\tau}$  where  $\tau$  is the expectation of delay, while in stochastic delay setting the upper bound can be  $\sqrt{K} + \tau$ . Therefore, a direct reduction from adversarial delay setting to stochastic delay setting will result in a suboptimal regret bound.
Low switching cost algorithm. Bandit problem with low switching cost has been widely studied in past decades. Cesa-Bianchi et al. (2013) showed an  $\tilde{\Theta}(K^{\frac{2}{3}})$  regret bound under adaptive adversaries and bounded memories. Perchet et al. (2016) proved a regret bound of  $\tilde{\Theta}(K^{\frac{1}{1-2^{1-M}}})$  for the two-armed bandit problem within  $M$  batches, and later Gao et al. (2019) extended their result to the general  $A$ -armed case. In RL, by doubling updates, the global switching cost is  $O(SAH\log_2(K))$  while keeping the regret  $\tilde{O}(\sqrt{SAKH^3})$  (Azar et al., 2017). Recently Zhang et al. (2022b) proposed a policy elimination algorithm that achieves  $\tilde{O}(\sqrt{SAKH^3})$  regret and  $O(H + \log_2\log_2(K))$  switching cost. Besides, Gao et al. (2021) generalized the problem to linear MDPs, and established a regret bound of  $\tilde{O}(\sqrt{d^3H^4K})$  with  $O(dH\log(K))$  global switching cost. Recent work Qiao et al. (2022) achieved  $O(HSA\log_2\log_2(K))$  switching cost and  $\tilde{O}(\text{poly}(S,A,H)\sqrt{K})$  regret with a computational inefficient algorithm. And Kong et al. (2021) consider MDP with general function approximation settings.
Markov Games. A standard framework to capture multi-agent RL is Markov games (also known as stochastic games) (Shapley, 1953). For two-player zero-sum Markov games, the goal is to solve for the Nash equilibrium (NE) (Nash, 1951). Recently, many works study this problem under the generative model (Zhang et al., 2020; Li et al., 2022), offline setting (Cui and Du, 2022b,a; Zhong et al., 2022; Xiong et al., 2022a; Yan et al., 2022), and online setting (Bai and Jin, 2020; Bai et al., 2020; Liu et al., 2021; Tian et al., 2021; Xie et al., 2020; Chen et al., 2021; Jin et al., 2022a; Xiong et al., 2022b; Huang et al., 2021). Our work is mostly related to Liu et al. (2021) and Xie et al. (2020). Specifically, Liu et al. (2021) studies online tabular two-player zero-sum MGs and proposes an algorithm with  $O(\sqrt{SABH^3K})$  regret. Xie et al. (2020) focused on online linear MGs provide an algorithm that enjoys  $O(\sqrt{d^3H^4K})$  regret. We also remark that Jin et al. (2022a); Xiong et al. (2022b); Huang et al. (2021) establish sharper regret bound for linear MGs but the algorithms therein are computationally inefficient. There is also a line of works (Liu et al., 2021; Jin et al., 2021;
Song et al., 2021; Zhong et al., 2021; Mao and Başar, 2022; Cui and Du, 2022b; Jin et al., 2022c; Daskalakis et al., 2022) studying multi-player general-sum MGs. Among these works, Jin et al. (2021); Song et al. (2021); Mao and Başar (2022) use V-learning algorithms to solve coarse correlated equilibrium (CCE). But all these algorithms can not handle delayed feedback nor are multi-batched algorithms that can be plugged into our framework. To this end, we present the first line of studies on multi-batched algorithms for MGs and provide a complete set of results for MGs with delayed feedback.
A very recent but independent work (Zhang et al., 2022a) also studies multi-agent reinforcement learning with delays. They assume that the reward is delayed, and aim to solve for coarse correlated equilibrium (CCE) in general-sum Markov games. Our work has several differences from Zhang et al. (2022a). First, they study the setting where the total delay is upper bounded by a certain threshold, while we focus on the setup with stochastic delay. Besides, their method seems only applicable for reward delay, while we allow the whole trajectory feedback to be delayed. Second, in the stochastic delay setting the delay-dependent term in their bound is worse than ours. Specifically, their bound scales as  $O(H^{3}\sqrt{SE[\tau]K})$ , while ours is  $O(H^{2}nSE[\tau])$  where  $n$  is the number of players. Finally, it seems that their results cannot be extended to the multi-agent RL with function approximation.
The objective of this section is to provide a unified view of the settings considered in this paper, i.e., bandits, Markov Decision Processes (MDPs) and multi-agent Markov Games. We consider a general framework for interactive decision making, which we refer to as Multi-agent Sequential Decision Making (MSDM).
Notations We use  $\Delta(\cdot)$  to represent the set of all probability distributions on a set. For  $n \in \mathbb{N}_+$ , we denote  $[n] = \{1, 2, \dots, n\}$ . We use  $O(\cdot), \Theta(\cdot), \Omega(\cdot)$  to denote the big-O, big-Theta, big-Omega notations. We use  $\widetilde{O}(\cdot)$  to hide logarithmic factors.
Multi-agent Sequential Decision Making A Multi-agent Sequential Decision Making problem can be represented as a tuple  $M = (n, S, \{A_i\}_{i=1}^n, H, \{p_h\}_{h=1}^H, s_1, \{r_h\}_{h=1}^H)$ , where  $n$  is the number of agents,  $S$  is the state space,  $A_i$  is the action space of player  $i$ ,  $H$  is the length of each episode and  $s_1$  is the initial state. We define  $A = \bigotimes_{i} A_i$  as the whole action space.
At each stage  $h$ , for every player  $i$ , every state-action pair  $(s, a_1, \dots, a_n)$  is characterized by a reward distribution with mean  $r_{i,h}(s, a_1, \dots, a_n)$  and support in  $[0, 1]$ , and a transition distribution  $p_h(\cdot | s, a)$  over next states. We denote by  $S = |\mathcal{S}|$  and  $A_i = |\mathcal{A}_i|$ .
The protocol proceeds in several episodes. At the beginning of each episode, the environment starts with initial state  $s_1$ . At each step  $h \in [1, H]$ , the agents observe the current state  $s_h$ , and take their actions  $(a_1, \dots, a_n)$  respectively. Then the environment returns a reward signal, and transits to next state  $s_{h+1}$ .
A (random) policy  $\pi_i$  of the  $i^{\mathrm{th}}$  player is a set of  $H$  maps  $\pi_i\coloneqq \{\pi_{i,h}:\Omega \times S\to \mathcal{A}_i\}_{h\in [H]}$ , where each  $\pi_{i,h}$  maps a random sample  $\omega$  from a probability space  $\Omega$  and state  $s$  to an action in  $\mathcal{A}_i$ . A joint (potentially correlated) policy is a set of policies  $\{\pi_i\}_{i = 1}^m$ , where the same random sample  $\omega$  is shared among all agents, which we denote as  $\pi = \pi_1\odot \pi_2\odot \ldots \odot \pi_m$ . We also denote  $\pi_{-i}$  as the joint policy excluding the  $i^{\mathrm{th}}$  player. For each stage  $h\in [H]$  and any state-action pair  $(s,a)\in S\times \mathcal{A}$ , the value function and Q-function of a policy  $\pi$  are defined as:
$$
Q _ {i, h} ^ {\pi} (s, a) = \mathbb {E} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {i, h ^ {\prime}} \mid s _ {h} = s, a _ {h} = a, \pi \right], \quad V _ {i, h} ^ {\pi} (s) = \mathbb {E} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {i, h ^ {\prime}} \mid s _ {h} = s, \pi \right].
$$
For each policy  $\pi$ , we define  $V_{H + 1}^{\pi}(s) = 0$  and  $Q_{H + 1}^{\pi}(s,a) = 0$  for all  $s\in S,a\in \mathcal{A}$ . Sometimes we omit the index  $i$  when it is clear from the context.
For the single agent case ( $n = 1$ ), the goal of the learner is to maximize the total reward it receives. There exists an optimal policy  $\pi^{\star}$  such that  $Q_{h}^{\star}(s,a) = Q_{h}^{\pi^{\star}}(s,a) = \max_{\pi} Q_{h}^{\pi}(s,a)$  satisfies the optimal Bellman equations
$$
Q _ {h} ^ {\star} (s, a) = r _ {h} (s, a) + \mathbb {E} _ {s ^ {\prime} \sim p _ {h} (s, a)} [ V _ {h + 1} ^ {\star} (s ^ {\prime}) ], \quad V _ {h} ^ {\star} (s) = \max _ {a \in \mathcal {A}} \{Q _ {h} ^ {\star} (s, a) \}, \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A}.
$$
Then the optimal policy is the greedy policy  $\pi_h^\star (s) = \arg \max_{a\in \mathcal{A}}\{Q_h^\star (s,a)\}$ . And we evaluate the performance of the learner through the notion of single-player regret, which is defined as
$$
\operatorname {R e g r e t} (K) = \sum_ {k = 1} ^ {K} \left(V _ {1} ^ {\star} - V _ {1} ^ {\pi_ {k}}\right) \left(s _ {1}\right). \tag {1}
$$
In the case of multi-player general-sum MGs, for any policy  $\pi_{-i}$ , the best response of the  $i^{\text{th}}$  player is defined as a policy of the  $i^{\text{th}}$  player which is independent of the randomness in  $\pi_{-i}$ , and achieves the highest value for herself conditioned on all other players deploying  $\pi_{-i}$ . In symbol, the best response is the maximizer of  $\max_{\pi_i'} V_{i,1}^{\pi_i' \times \pi_{-i}}(s_1)$  whose value we also denote as  $V_{i,1}^{\dagger, \pi_{-i}}(s_1)$  for simplicity.
For general-sum MGs, we aim to learn the Coarse Correlated Equilibrium (CCE), which is defined as a joint (potentially correlated) policy where no player can increase her value by playing a different independent strategy. In symbol,
Definition 1 (Coarse Correlated Equilibrium). A joint policy  $\pi$  is a CCE if  $\max_{i\in [m]}(V_{i,1}^{\dagger,\pi_{-i}} - V_{i,1}^{\pi})$  ( $s_1 = 0$ ). A joint policy  $\pi$  is a  $\epsilon$ -approximate CCE if  $\max_{i\in [m]}(V_{i,1}^{\dagger,\pi_{-i}} - V_{i,1}^{\pi})(s_1) \leq \epsilon$ .
For any algorithm that outputs policy  $\hat{\pi}^k$  at episode  $k$ , we measure the performance of the algorithm using the following notion of regret:
$$
\operatorname {R e g r e t} (K) = \sum_ {k = 1} ^ {K} \max  _ {j} \left(V _ {j, h} ^ {\dagger , \hat {\pi} _ {- j, h} ^ {k}} - V _ {j, h} ^ {\hat {\pi} _ {h} ^ {k}}\right) \left(s _ {1}\right). \tag {2}
$$
One special case of multi-player general-sum MGs is two-player zero-sum MGs, in which there are only two players and the reward satisfies  $r_{2,h} = -r_{1,h}$ . In zero-sum MGs, there exist policies that are the best responses to each other. We call these optimal strategies the Nash equilibrium of the Markov game, which satisfies the following minimax equation:
$$
\sup  _ {\mu} \inf  _ {\nu} V _ {h} ^ {\mu , \nu} (s) = V _ {h} ^ {\mu^ {*}, \nu^ {*}} (s) = \inf  _ {\nu} \sup  _ {\mu} V _ {h} ^ {\mu , \nu} (s).
$$
We measure the suboptimality of any pair of general policies  $(\hat{\mu},\hat{\nu})$  using the gap between their performance and the performance of the optimal strategy (i.e., Nash equilibrium) when playing against the best responses respectively:
$$
V _ {1} ^ {\dagger , \hat {\nu}} (s _ {1}) - V _ {1} ^ {\hat {\mu}, \dagger} (s _ {1}) = \left[ V _ {1} ^ {\dagger , \hat {\nu}} (s _ {1}) - V _ {1} ^ {\star} (s _ {1}) \right] + \left[ V _ {1} ^ {\star} (s _ {1}) - V _ {1} ^ {\hat {\mu}, \dagger} (s _ {1}) \right].
$$
Let  $(\mu^k,\nu^k)$  denote the policies deployed by the algorithm in the  $k^{\mathrm{th}}$  episode. The learning goal is to minimize the two-player regret of  $K$  episodes, defined as
$$
\operatorname {R e g r e t} (K) = \sum_ {k = 1} ^ {K} \left(V _ {1} ^ {\dagger , \nu^ {k}} - V _ {1} ^ {\mu^ {k}, \dagger}\right) \left(s _ {1}\right). \tag {3}
$$
One can see that (3) is a natural reduction of (2) from general-sum games to zero-sum games.
MSDM with Delayed Feedback We focus on the case where there is delay between playing an episode and observing the sequence of states, actions and rewards sampled by the agent; we refer to this sequence as feedback throughout this paper.
We assume that the delays are stochastic, i.e. we introduce a random variable for the delay between playing the  $k$ -th episode and observing the feedback of the  $k$ -th episode, which we denote by  $\tau_{k}$ . We assume the delays are i.i.d:
Assumption 1. The delays  $\{\tau_k\}_{k=1}^k$  are positive, independent and identically distributed random variables:  $\tau_k \stackrel{i.i.d.}{\sim} f_\tau(\cdot)$ . We denote the expectation of  $\tau$  as  $\mathbb{E}[\tau]$  (which may be infinite), and denote the quantile function of  $\tau$  as  $d_\tau(q)$ , i.e.
$$
d _ {\tau} (q) = \min  \left\{\gamma \in \mathbb {N} \mid \Pr [ \tau \leq \gamma ] \geq q \right\}. \tag {4}
$$
When delay exists, the feedback associated with an episode does not return immediately. Instead, the feedback of the  $k$ -th episode cannot be observed until the  $k + \tau_{k}$ -th episode ends.
Sometimes in order to control the scale and distribution of the delay, we make the assumption that the delays are  $(v,b)$  subexponential random variables. This is a standard assumption used in previous works Howson et al. (2021), though it is not always necessary in our paper.
Assumption 2. The delays are non-negative, independent, and identically distributed  $(v,b)$ -subexponential random variables. That is, their moment generating function satisfies the following inequality
$$
\mathbb {E} \left[ \exp \left(\gamma \left(\tau_ {t} - \mathbb {E} [ \tau_ {t} ]\right)\right) \right] \leq \exp \left(\frac {1}{2} v ^ {2} \gamma^ {2}\right) \tag {5}
$$
for some  $v$  and  $b$ , and all  $|\gamma| \leq 1 / b$ .
3 Multi-batched Algorithm for MSDMs
In this section we mainly discuss multi-batched algorithms for MSDMs. This section aims to provide a formal definition of multi-batched algorithm, and give some examples of multi-batched algorithm for the ease of understanding.
Formally, a multi-batched algorithm consists of several batches. At the beginning of the  $m^{\mathrm{th}}$  batch, the algorithm outputs a policy sequence  $\pi^{m} = (\pi_{1}^{m},\pi_{2}^{m},\dots,\pi_{l^{m}}^{m})$  for this batch using data collected in previous batches, along with a stopping criteria  $SC$ , which is a mapping from a dataset to a boolean value. Then the agent executes the policy sequence  $\pi^{m}$  for several episodes and collects feedback to the current dataset  $\mathcal{D}$ , until the stopping criteria are satisfied (i.e.  $SC(\mathcal{D}) = 1$ ). If the length of the  $m^{\mathrm{th}}$  batch exceeds  $l_{m}$ , then we simply run the policy sequence cyclically, which means in the  $k^{\mathrm{th}}$  episode we run policy  $\pi_{k\mod l^{m}}^{m}$ . Algorithm 1 describes a protocol of multi-batched algorithm.
Algorithm 1 Protocol of Multi-batched Algorithm
1: Initialize dataset  $\mathcal{D}^0 = \emptyset$    
2: for batch  $m = 1,\dots,M$  do   
3:  $\mathcal{D} = \emptyset$    
4:  $k = 0$    
5: Calculate a policy sequence  $\pi^m = (\pi_1^m,\pi_2^m,\dots,\pi_{l^m}^m)$  using previous data  $\mathcal{D}^{m - 1}$  , and a stopping criteria SC   
6: while  $SC(\mathcal{D}) = 0$  do   
7:  $k\gets k + 1$    
8: In episode  $k$  , execute  $\pi_{k\bmod l^{m}}^{m}$  and collect trajectory feedback  $o^k$    
9:  $\mathcal{D} = \mathcal{D}\cup \{o^k\}$    
10: end while   
11: Update dataset  $\mathcal{D}^m = \mathcal{D}^{m - 1}\cup \mathcal{D}$    
12: end for   
13: Output: A policy sequence  $\{\pi^m\}$
Note that the major requirement is that the policy to be executed during the batch should be calculated at the beginning of the batch. Therefore, algorithms that calculate the policy at the beginning of each episode do not satisfy this requirement. For example, UCB algorithms in bandits (Lattimore and Szeptsvári, 2020) and UCBVI (Azar et al., 2017) are not multi-batched algorithms.
Relation with low-switching cost algorithm Our definition of multi-batched algorithm is similar to that of low-switching cost algorithm, but is more general.
For any algorithm, the switching cost is the number of policy changes in the running of the algorithm in  $K$  episodes, namely:
$$
N _ {\text {s w i t c h}} \triangleq \sum_ {k = 1} ^ {K - 1} \mathbb {I} \left\{\pi_ {k} \neq \pi_ {k + 1} \right\}.
$$