Title: A Reduction-based Framework for Sequential Decision Making with Delayed Feedback
Venue: NeurIPS 2023
Accepted: yes
Keywords: sequential decision making

delay

reinforcement learning
Abstract: We study stochastic delayed feedback in general single-agent and multi-agent sequential decision making, which includes bandits, single-agent Markov decision processes (MDPs), and Markov games (MGs). We propose a novel reduction-based framework, which turns any multi-batched algorithm for sequential decision making with instantaneous feedback into a sample-efficient algorithm that can handle stochastic delays in sequential decision making. By plugging different multi-batched algorithms into our framework, we provide several examples demonstrating that our framework not only matches or improves existing results for bandits, tabular MDPs, and tabular MGs, but also provides the first line of studies on delays in sequential decision making with function approximation. In summary, we provide a complete set of sharp results for single-agent and multi-agent sequential decision making with delayed feedback.

A Reduction-based Framework for Sequential Decision Making with Delayed Feedback

Abstract

We study stochastic delayed feedback in general sequential decision-making problems, which include bandits, single-agent Markov decision processes (MDPs), and Markov games (MGs). We propose a novel reduction-based framework, which turns any multi-batched algorithm for sequential decision making with instantaneous feedback into a sample-efficient algorithm that can handle stochastic delays in sequential decision-making problems. By plugging different multi-batched algorithms into our framework, we provide several examples demonstrating that our framework not only matches or improves existing results for bandits, tabular MDPs, and tabular MGs, but also provides the first line of studies on delays in sequential decision making with function approximation. In summary, we provide a complete set of sharp results for single-agent and multi-agent sequential decision-making problems with delayed feedback.

1 Introduction

Delayed feedback is a common and fundamental problem in sequential decision making (Sutton and Barto, 2018; Lattimore and Szepesvári, 2020; Zhang et al., 2021a). Taking the recommendation system as an example, delayed feedback is an inherent part of this problem. Specifically, the learner adjusts her recommendation after receiving users' feedback, which may take a random delay after the recommendation is issued. More examples include but are not limited to robotics (Mahmood et al., 2018) and video steaming (Changuel et al., 2012). Furthermore, the delayed feedback issue is more serious in multi-agent systems (Chen et al., 2020) since observing the actions of other agents' may happen with a varying delay.

Although handling delayed feedback is a prominent challenge in practice, the theoretical understanding of delays in sequential decision making is limited. Even assuming the delays are stochastic (delays are sampled from a fixed distribution), the nearly minimax optimal result is only obtained by works on multi-armed bandits and contextual bandits (Agarwal and Duchi, 2011; Dudik et al., 2011; Joulani et al., 2013; Vernade et al., 2020, 2017; Gael et al., 2020). For more complex problems such as linear bandits (Lancewicki et al., 2021) and tabular reinforcement learning (RL) (Howson et al., 2022), the results are suboptimal. Meanwhile, to the best of our knowledge, there are no theoretical guarantees in RL with function approximation and multi-agent RL settings. Hence, we focus on stochastic delays and aim to answer the following two questions in this work:

Can we provide sharper regret bounds for basic models such as linear bandits and tabular RL?

̸

̸

̸

2. Can we handle delays in (multi-agent) sequential decision making in the context of function approximation?

We answer these two questions affirmatively by proposing a reduction-based framework for stochastic delays in both single-agent and multi-agent sequential decision making. Our contributions are summarized below.

Our Contributions. Our main contribution is proposing a new reduction-based framework for both single-agent and multi-agent sequential decision making with stochastic delayed feedback. The proposed framework can convert any multi-batched algorithm (cf. Section 3) for sequential decision making with instantaneous feedback to a provably efficient algorithm that is capable of handling stochastic delayed feedback in sequential decision making. Unlike previous works which require case by case algorithm design and analysis, we provide a unified framework to solve a wide range of problems all together.

Based on this framework, we obtain a complete set of results for single-agent and multi-agent sequential decision making with delayed feedback. In specific, our contributions can be summarized as follows:

New framework: We propose a generic algorithm that can be integrated with any multi-batched algorithm in a black-box fashion. Meanwhile, we provide a unified theoretical analysis for the proposed generic algorithm.

Improved results and new results: By applying our framework to different settings, we obtain state-of-the-art regret bounds for multi-armed bandits, and derive sharper results for linear bandits and tabular RL, which significantly improve existing results (Lancewicki et al., 2021; Howson et al., 2022). We also provide the first result for single-agent RL with delayed feedback in the context of linear or even general function approximation;

New algorithms: We show that delayed feedback in Markov games (MGs) can also be handled by our framework. By plugging several new proposed multi-batched algorithms for MGs into our framework, we not only improve recent results on tabular MGs (Zhang et al., 2022a), but

also present a new result for linear MGs. As a byproduct, we provide the first line of study on multi-batched algorithms for MGs, which might be of independent interest.

Our results and comparisons with existing works are summarized in Table 1.

1.1 Related Works

Bandit/MDP with delayed feedback. Stochastically delayed feedback has been studied a lot in the multi-armed bandit and contextual bandit problems (Agarwal and Duchi, 2011; Dudik et al., 2011; Joulani et al., 2013; Vernade et al., 2020, 2017; Gael et al., 2020; Huang et al., 2023). A recent work (Lancewicki et al., 2021) shows that under stochastic delay, elimination algorithm performs better than UCB algorithm, and gives tight upper and lower bounds for multi-armed bandit setting. Howson et al. (2021) solves the linear bandit setting by adapting LinUCB algorithm. The concurrent work of Vakili et al. (2023) provides a bound for the kernel bandit setting with delayed feedback. Our work does not consider the kernel setting, but we conjecture that our framework can handle this problem by designing a multi-batched algorithm for kernel bandits.

In RL, despite practical importance, there is limited literature on stochastical delay in RL. Lancewicki et al. (2022) and Jin et al. (2022b) considered adversarial delays. Their regret depends on the sum of the delays, the number of states and the number of steps per episode. However, when reducing their results to stochastic delay, the bounds will be too loose. Howson et al. (2022) considered stochastic delay in tabular MDP, but their bound is not tight. As far as we know, there is no work considering linear MDP and general function approximation with delayed feedback.

Another line of work considers adversarially delayed feedback (Lancewicki et al., 2022; Jin et al., 2022b; Quanrud and Khashabi, 2015; Thune et al., 2019; Bistritz et al., 2019; Zimmert and Seldin, 2020; Ito et al., 2020; Gyorgy and Joulani, 2021; Van Der Hoeven and Cesa-Bianchi, 2022). In the adversarial setting the state-of-the-art result is of form √ K + D , where K is the number of the episodes and D is the total delay. Note that when adapting their result to the stochastic setting, the bound becomes √ K + Kτ where τ is the expectation of delay, while in stochastic delay setting the upper bound can be √ K + τ . Therefore, a direct reduction from adversarial delay setting to stochastic delay setting will result in a suboptimal regret bound.

Low switching cost algorithm. Bandit problem with low switching cost has been widely studied in past decades. Cesa-Bianchi et al. (2013) showed an ˜ Θ( K 2 3 ) regret bound under adaptive adversaries and bounded memories. Perchet et al. (2016) proved a regret bound of ˜ Θ( K 1 1 -2 1 -M ) for the two-armed bandit problem within M batches, and later Gao et al. (2019) extended their result to the general A -armed case. In RL, by doubling updates, the global switching cost is O ( SAH log 2 ( K )) while keeping the regret ˜ O ( √ SAKH 3 ) (Azar et al., 2017). Recently Zhang et al. (2022b) proposed a policy elimination algorithm that achieves ˜ O ( √ SAKH 3 ) regret and O ( H +log 2 log 2 ( K )) switching cost. Besides, Gao et al. (2021) generalized the problem to linear MDPs, and established a regret bound of ˜ O ( √ d 3 H 4 K ) with O ( dH log( K )) global switching cost. Recent work Qiao et al. (2022) achieved O ( HSA log 2 log 2 ( K )) switching cost and ˜ O (poly( S, A, H ) √ K ) regret with a computational inefficient algorithm. And Kong et al. (2021) consider MDP with general function approximation settings.

Markov Games. A standard framework to capture multi-agent RL is Markov games (also known as stochastic games) (Shapley, 1953). For two-player zero-sum Markov games, the goal is to solve for the Nash equilibrium (NE) (Nash, 1951). Recently, many works study this problem under the generative model (Zhang et al., 2020; Li et al., 2022), offline setting (Cui and Du, 2022b,a; Zhong et al., 2022; Xiong et al., 2022a; Yan et al., 2022), and online setting (Bai and Jin, 2020; Bai et al., 2020; Liu et al., 2021; Tian et al., 2021; Xie et al., 2020; Chen et al., 2021; Jin et al., 2022a; Xiong et al., 2022b; Huang et al., 2021). Our work is mostly related to Liu et al. (2021) and Xie et al. (2020). Specifically, Liu et al. (2021) studies online tabular two-player zero-sum MGs and proposes an algorithm with O ( √ SABH 3 K ) regret. Xie et al. (2020) focused on online linear MGs provide an algorithm that enjoys O ( √ d 3 H 4 K ) regret. We also remark that Jin et al. (2022a); Xiong et al. (2022b); Huang et al. (2021) establish sharper regret bound for linear MGs but the algorithms therein are computationally inefficient. There is also a line of works (Liu et al., 2021; Jin et al., 2021;

Song et al., 2021; Zhong et al., 2021; Mao and Ba¸ sar, 2022; Cui and Du, 2022b; Jin et al., 2022c; Daskalakis et al., 2022) studying multi-player general-sum MGs. Among these works, Jin et al. (2021); Song et al. (2021); Mao and Ba¸ sar (2022) use V-learning algorithms to solve coarse correlated equilibrium (CCE). But all these algorithms can not handle delayed feedback nor are multi-batched algorithms that can be plugged into our framework. To this end, we present the first line of studies on multi-batched algorithms for MGs and provide a complete set of results for MGs with delayed feedback.

A very recent but independent work (Zhang et al., 2022a) also studies multi-agent reinforcement learning with delays. They assume that the reward is delayed, and aim to solve for coarse correlated equilibrium (CCE) in general-sum Markov games. Our work has several differences from Zhang et al. (2022a). First, they study the setting where the total delay is upper bounded by a certain threshold, while we focus on the setup with stochastic delay. Besides, their method seems only applicable for reward delay, while we allow the whole trajectory feedback to be delayed. Second, in the stochastic delay setting the delay-dependent term in their bound is worse than ours. Specifically, their bound scales as O ( H 3 √ S E [ τ ] K ) , while ours is O ( H 2 nS E [ τ ]) where n is the number of players. Finally, it seems that their results cannot be extended to the multi-agent RL with function approximation.

2 Preliminary

The objective of this section is to provide a unified view of the settings considered in this paper, i.e., bandits, Markov Decision Processes (MDPs) and multi-agent Markov Games. We consider a general framework for interactive decision making, which we refer to as Multi-agent Sequential Decision Making (MSDM).

Notations We use ∆( · ) to represent the set of all probability distributions on a set. For n ∈ N + , we denote [ n ] = { 1 , 2 , . . . , n } . We use O ( · ) , Θ( · ) , Ω( · ) to denote the big-O, big-Theta, big-Omega notations. We use ˜ O ( · ) to hide logarithmic factors.

Multi-agent Sequential Decision Making AMulti-agent Sequential Decision Making problem can be represented as a tuple M = ( n, S , {A i } n i =1 , H, { p h } H h =1 , s 1 , { r h } H h =1 ) , where n is the number of agents, S is the state space, A i is the action space of player i , H is the length of each episode and s 1 is the initial state. We define A = ⊗ i A i as the whole action space.

At each stage h , for every player i , every state-action pair ( s, a 1 , ..., a n ) is characterized by a reward distribution with mean r i,h ( s, a 1 , ..., a n ) and support in [0 , 1] , and a transition distribution p h ( ·| s, a ) over next states. We denote by S = |S| and A i = |A i | .

The protocol proceeds in several episodes. At the beginning of each episode, the environment starts with initial state s 1 . At each step h ∈ [1 , H ] , the agents observe the current state s h , and take their actions ( a 1 , ..., a n ) respectively. Then the environment returns a reward signal, and transits to next state s h +1 .

A (random) policy π i of the i th player is a set of H maps π i := { π i,h : Ω ×S → A i } h ∈ [ H ] , where each π i,h maps a random sample ω from a probability space Ω and state s to an action in A i . A joint (potentially correlated) policy is a set of policies { π i } i m =1 , where the same random sample ω is shared among all agents, which we denote as π = π 1 ⊙ π 2 ⊙ . . . ⊙ π m . We also denote π -i as the joint policy excluding the i th player. For each stage h ∈ [ H ] and any state-action pair ( s, a ) ∈ S × A , the value function and Q-function of a policy π are defined as:

For each policy π , we define V π H +1 ( s ) = 0 and Q π H +1 ( s, a ) = 0 for all s ∈ S , a ∈ A . Sometimes we omit the index i when it is clear from the context.

For the single agent case ( n = 1 ), the goal of the learner is to maximize the total reward it receives. There exists an optimal policy π ⋆ such that Q ⋆ h ( s, a ) = Q π ⋆ h ( s, a ) = max π Q π h ( s, a ) satisfies the optimal Bellman equations

Then the optimal policy is the greedy policy π ⋆ h ( s ) = arg max a ∈A { Q ⋆ h ( s, a ) } . And we evaluate the performance of the learner through the notion of single-player regret, which is defined as

In the case of multi-player general-sum MGs, for any policy π -i , the best response of the i th player is defined as a policy of the i th player which is independent of the randomness in π -i , and achieves the highest value for herself conditioned on all other players deploying π -i . In symbol, the best response is the maximizer of max π ′ i V π ′ i × π -i i, 1 ( s 1 ) whose value we also denote as V † ,π -i i, 1 ( s 1 ) for simplicity.

For general-sum MGs, we aim to learn the Coarse Correlated Equilibrium (CCE), which is defined as a joint (potentially correlated) policy where no player can increase her value by playing a different independent strategy. In symbol,

Definition 1 (Coarse Correlated Equilibrium) . A joint policy π is a CCE if max i ∈ [ m ] ( V † ,π -i i, 1 -V π i, 1 ) ( s 1 ) = 0. A joint policy π is a ϵ -approximate CCE if max i ∈ [ m ] ( V † ,π -i i, 1 -V π i, 1 )( s 1 ) ≤ ϵ .

For any algorithm that outputs policy ˆ π k at episode k , we measure the performance of the algorithm using the following notion of regret:

One special case of multi-player general-sum MGs is two-player zero-sum MGs, in which there are only two players and the reward satisfies r 2 ,h = -r 1 ,h . In zero-sum MGs, there exist policies that are the best responses to each other. We call these optimal strategies the Nash equilibrium of the Markov game, which satisfies the following minimax equation:

We measure the suboptimality of any pair of general policies (ˆ µ, ˆ ν ) using the gap between their performance and the performance of the optimal strategy (i.e., Nash equilibrium) when playing against the best responses respectively:

Let ( µ k , ν k ) denote the policies deployed by the algorithm in the k th episode. The learning goal is to minimize the two-player regret of K episodes, defined as

One can see that (3) is a natural reduction of (2) from general-sum games to zero-sum games.

MSDMwith Delayed Feedback We focus on the case where there is delay between playing an episode and observing the sequence of states, actions and rewards sampled by the agent; we refer to this sequence as feedback throughout this paper.

We assume that the delays are stochastic, i.e. we introduce a random variable for the delay between playing the k -th episode and observing the feedback of the k -th episode, which we denote by τ k . We assume the delays are i.i.d:

Assumption 1. The delays { τ k } k k =1 are positive, independent and identically distributed random variables: τ k i.i.d. ∼ f τ ( · ) . We denote the expectation of τ as E [ τ ] (which may be infinite), and denote the quantile function of τ as d τ ( q ) , i.e.

When delay exists, the feedback associated with an episode does not return immediately. Instead, the feedback of the k -th episode cannot be observed until the k + τ k -th episode ends.

Sometimes in order to control the scale and distribution of the delay, we make the assumption that the delays are ( v, b ) subexponential random variables. This is a standard assumption used in previous works Howson et al. (2021), though it is not always necessary in our paper.

Assumption 2. The delays are non-negative, independent, and identically distributed ( v, b ) - subexponential random variables. That is, their moment generating function satisfies the following inequality

for some v and b , and all | γ | ≤ 1 /b .

3 Multi-batched Algorithm for MSDMs

In this section we mainly discuss multi-batched algorithms for MSDMs. This section aims to provide a formal definition of multi-batched algorithm, and give some examples of multi-batched algorithm for the ease of understanding.

Formally, a multi-batched algorithm consists of several batches. At the beginning of the m th batch, the algorithm outputs a policy sequence π m = ( π m 1 , π m 2 , ..., π l m m ) for this batch using data collected in previous batches, along with a stopping criteria SC , which is a mapping from a dataset to a boolean value. Then the agent executes the policy sequence π m for several episodes and collects feedback to the current dataset D , until the stopping criteria are satisfied (i.e. SC ( D ) = 1 ). If the length of the m th batch exceeds l m , then we simply run the policy sequence cyclically, which means in the k th episode we run policy π m k mod l m . Algorithm 1 describes a protocol of multi-batched algorithm.

Algorithm 1 Protocol of Multi-batched Algorithm

, and a

Note that the major requirement is that the policy to be executed during the batch should be calculated at the beginning of the batch. Therefore, algorithms that calculate the policy at the beginning of each episode do not satisfy this requirement. For example, UCB algorithms in bandits (Lattimore and Szepesvári, 2020) and UCBVI (Azar et al., 2017) are not multi-batched algorithms.

Relation with low-switching cost algorithm Our definition of multi-batched algorithm is similar to that of low-switching cost algorithm, but is more general.

For any algorithm, the switching cost is the number of policy changes in the running of the algorithm in K episodes, namely:

̸

Most algorithms with N switch low-switching cost are multi-batched algorithms with N switch batches. For example, the policy elimination algorithm proposed in Zhang et al. (2022b) is a multi-batched

algorithm with 2 H +loglog K batches. And the algorithm for linear MDP with low global switching cost in Gao et al. (2021) is a multi-batched algorithm with O ( dH log K ) batches. On the other hand, the difference between batches and switching cost is that we do not require the policy to be the same during a batch. Therefore, a multi-batched algorithm with N batches may not be an algorithm with N switching cost. For example, the Phase Elimination Algorithm in linear bandit with finite arms setting (Lattimore and Szepesvári, 2020) is a multi-batched algorithm with O (log K ) batches, but the switching cost is O ( A log K ) which is much larger.

4 A Framework for Sequential Decision Making with Delayed Feedback

In this section, we introduce our framework for sequential decision making problems with stochastic delayed feedback. We first describe our framework, which converts any multi-batched algorithm to an algorithm for MSDM with delayed feedback. Then we provide theoretical guarantees for the performance of such algorithms.

Before formally introducing our framework, we first give an intuitive explanation of why multibatched algorithms work for delayed settings. Recall that Lancewicki et al. (2021) gives a lower bound showing that UCB-type algorithms are suboptimal in multi-armed bandits with stochastic delay (see Theorem 1 in Lancewicki et al. (2021)). The main idea of the lower bound is that UCB algorithm will keep pulling a suboptimal arm until the delayed feedback arrives and the policy is updated. Therefore, if the policy updates too frequently, then the algorithm has to spend more time waiting for the update, and the effect of delay will be greater. This observation motivates us to consider multi-batched algorithms, which have a low frequency of policy updating.

For any sequential decision making problem, assume that we have a multi-batched algorithm ALG for the problem without delay. Then in the existence of delay, we can still run ALG . The difference is, in each batch of the algorithm, we run some extra steps until we observe enough feedback so that the stopping criteria are satisfied. For example, if the delay τ is a constant and the stopping criteria is that the policy π k is executed for a pre-defined number of times t k , then we only need to run π k for t k + τ times. After the extra steps are finished, we can collect t k delayed observations, which meet the original stopping criteria. Then we can continue to the next batch. Algorithm 2 gives a formal description.

Algorithm 2 Multi-batched Algorithm With Delayed Feedback

For the performance of Algorithm 2, we have the following theorem, which relates the regret of delayed MSDM problems with the regret of multi-batched algorithms in the undelayed environment.

Theorem 1. Assume that in the undelayed environment, we have a multi-batched algorithm with N b batches, and the regret of the algorithm in K episodes can be upper bounded by ˜ Regret( K ) with probability at least 1 -δ . Then in the delayed feedback case, with probability at least 1 -δ , the regret can be upper bounded by

for any q ∈ (0 , 1) . In addition, if the delays satisfy Assumption 2, then with probability at least 1 -δ the regret can be upper bounded by

where C τ = min { √ 2 v 2 log(3 KH/ (2 δ )) , 2 b log(3 KH/ (2 δ )) } is a problem-independent constant.

The proof is in Appendix A. The main idea is simple: in each batch, we can control the number of extra steps when waiting for the original algorithm to collect enough delayed feedback. And the extra regret can be bounded by the product of the number of batches and the number of extra steps per batch. Note that in Eq. (6) we do not need Assumption 2. This makes (6) more general.

Applying Theorem 1 to different settings, we will get various types of results. For example, in multi-armed bandit setting, using the Batched Successive Elimination (BaSE) algorithm in Gao et al. (2019), we can get a O ( 1 q A log( K ) + d τ ( q ) log( K ) upper bound from (6), which recovers the SOTA result in Lancewicki et al. (2021) up to a log K factor in the delay-dependent term. We leave the details to Appendix C, and list the main results in Table 1.

5 Results for Markov Games

As an application, we study Markov games with delayed feedback using our framework. We first develop new multi-batched algorithms for Markov games, then combine them with our framework to obtain regret bounds for the delayed setting.

5.1 Tabular Zero-Sum Markov Game

First we study the tabular zero-sum Markov game setting. The algorithm is formally described in Algorithm 4 in Appendix D.

Our algorithm can be decomposed into two parts: an estimation step and a policy update step. The estimation step largely follows the method in Liu et al. (2021). For each ( h, s, a, b ) , we maintain an optimistic estimation ¯ Q k h ( s, a, b ) and a pessimistic estimation Q k h ( s, a, b ) of the underlying Q ⋆ h ( s, a, b ) , by value iteration with bonus using the empirical estimate of the transition ˆ P . And we compute the Coarse Correlated Equilibrium (CCE) policy π of the estimated value functions Xie et al. (2020). A CCE always exists, and can be computed efficiently.

In the policy update step, we choose the CCE policy w.r.t. the current estimated value function. Note that we use the doubling trick to control the number of updates: we only update the policy when there exists ( h, s, a, b ) such that the visiting count N h ( s, a, b ) is doubled. This update scheme ensures that the batch number is small.

For the regret bound of Algorithm 4, we have the following theorem.

Theorem 2. For any p ∈ (0 , 1] , letting ι = log( SABT/p ) , then with probability at least 1 -p , the regret of Algorithm 4 satisfies

and the batch number is at most O ( HSAB log K ) .

The proof of Theorem 4 is similar to Theorem 4 in Liu et al. (2021), but the main difference is we have to deal with the sum of the Bernstein-type bonus under the doubling framework. To deal with this we borrow ideas from Zhang et al. (2021b). The whole proof is in Appendix D.

Adapting the algorithm to the delayed feedback setting using our framework, we have the following corollary.

Corollary 1. For tabular Markov game with delayed feedback, running Algorithm 2 with ALG being Algorithm 4, we can upper bound the regret by

for any q ∈ (0 , 1) . In addition, if the delays satisfy Assumption 2, the regret can be upper bounded by Regret( K ) ≤ O ( √ H 3 SABKι + H 3 S 2 ABι 2 + H 2 SAB E [ τ ] log K ) .

5.2 Linear Zero-Sum Markov Game

We consider the linear zero-sum Markov game setting, where the reward and transition have a linear structure.

Assumption 3. For each ( x, a, b ) ∈ S × A × B and h ∈ [ H ] , we have

where ϕ : S × A × A → R d is a known feature map, θ h ∈ R d is an unknown vector and µ h = ( µ ( i ) h ) i ∈ [ d ] is a vector of d unknown (signed) measures on S . We assume that ∥ ϕ ( · , · , · ) ∥ ≤ 1 , ∥ θ h ∥ ≤ √ d and ∥ µ h ( S ) ∥ ≤ √ d for all h ∈ [ H ] , where ∥ · ∥ is the vector ℓ 2 norm.

The formal algorithm is in Algorithm 5 in Appendix D.2. It is also divided into two parts. The estimation step is similar to that in Xie et al. (2020), where we calculate an optimistic estimation ¯ Q k h ( s, a, b ) and a pessimistic estimation Q k h ( s, a, b ) of the underlying Q ⋆ h ( s, a, b ) by least square value iteration. And we compute the CCE policy π of the estimated value functions.

In the policy update step, we update the policy to the CCE policy w.r.t. the current estimated value function. Note that instead of updating the policy in every episode, we only update it when there exists h ∈ [ H ] such that det(Λ k h ) > η · det(Λ h ) . This condition implies that we have collected enough information at one direction. This technique is similar to Gao et al. (2021); Wang et al. (2021).

For the regret bound of Algorithm 5, we have the following theorem.

Theorem 3. In Algorithm 5, let η = 2 . Then we have

And the batch number of Algorithm 5 is dH log 2 log ( 1 + K λ ) .

Proof.

Adapting the algorithm to the delayed feedback setting using our framework, we have the following corollary.

Corollary 2. For linear Markov game with delayed feedback, running Algorithm 2 with ALG being Algorithm 5, we can upper bound the regret by

for any q ∈ (0 , 1) . In addition, if the delays satisfy Assumption 2, the regret can be upper bounded by

5.3 Tabular General-Sum Markov Game

In this section, we introduce the multi-batched version of V-learning (Jin et al., 2021; Mao and Ba¸ sar, 2022; Song et al., 2021) for general-sum Markov games. Our goal is to minimize the regret (2). The algorithm is formally described in Algorithm 6 in Appendix D.

The multi-batched V-learning algorithm maintains a value estimator V h ( s ) , a counter N h ( s ) , and a policy π h ( · | s ) for each s and h . We also maintain S × H different adversarial bandit algorithms. At step h in episode k , the algorithm is divided into three steps: policy execution, V -value update, and policy update. In policy execution step, the algorithm takes action a h according to π ˜ k h , and observes reward r h and the next state s h +1 , and updates the counter N h ( s h ) . Note that π ˜ k h is updated only when the visiting count of some state N h ( s ) doubles. Therefore it ensures a low batch number.

In the V -value update step, we update the estimated value function by

where the learning rate is defined as

and β ( t ) is the bonus to promote optimism.

In the policy update step, the algorithm feeds the action a h and its "loss" H -r h + V h +1 ( s h +1 ) H to the ( s h , h ) th adversarial bandit algorithm. Then it receives the updated policy π h ( · | s h ) .

For two-player general sum Markov games, we let the two players run Algorithm 6 independently. Each player uses her own set of bonus that depends on the number of her actions. If we choose the adversarial bandit algorithm as Follow The Regularized Leader (FTRL) algorithm (Lattimore and Szepesvári, 2020), we have the following theorem for the regret of multi-batched V-learning.

Theorem 4. Suppose we choose the adversarial bandit algorithm as FTRL. For any δ ∈ (0 , 1) and K ∈ N , let ι = log( HSAK/δ ) . Choose learning rate α t and bonus { β ( t ) } K t =1 as β ( t ) = c · √ H 3 Aι/t so that ∑ t i =1 α i t β ( i ) = Θ( √ H 3 Aι/t ) for any t ∈ [ K ] , where A = max i A i Then, with probability at least 1 -δ , after running Algorithm 6 for K episodes, we have

And the batch number is O ( nHS log K ) , where n is the number of players.

Proof.

Finally, adapting the algorithm to the delayed feedback setting using our framework, we have the following corollary.

Corollary 3. For tabular general-sum Markov game with delayed feedback, running Algorithm 2 with ALG being Algorithm 6, we can upper bound the regret by

for any q ∈ (0 , 1) . In addition, if the delays satisfy Assumption 2, the regret can be upper bounded by

Proof. This corollary is directly implied by Theorems 1 and 4.

6 Conclusion and Future Work

In this paper we study delayed feedback in general multi-agent sequential decision making. We propose a novel reduction-based framework, which turns any multi-batched algorithm for sequential decision making with instantaneous feedback into a sample-efficient algorithm that can handle stochastic delays in sequential decision making. Based on this framework, we obtain a complete set of results for (multi-agent) sequential decision making with delayed feedback.

Our work has also shed light on future works. First, it remains unclear whether the result is tight for MDP and Markov game with delayed feedback. Second, it is possible that we can derive a multi-batched algorithm for tabular Markov game with a smaller batch number, since in tabular MDP, Zhang et al. (2022b) gives an algorithm with O ( H +log 2 log 2 K ) batch number which is independent of S . It is not clear whether we can achieve similar results in MDP for Markov games. We believe a tighter result is possible in this setting with a carefully designed multi-batched algorithm.

References

Abbasi-Yadkori, Y., Pál, D., and Szepesvári, C. (2011). Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24.

Agarwal, A. and Duchi, J. C. (2011). Distributed delayed stochastic optimization. Advances in neural information processing systems , 24.

Azar, M. G., Osband, I., and Munos, R. (2017). Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR.

Bai, Y. and Jin, C. (2020). Provable self-play algorithms for competitive reinforcement learning. In International conference on machine learning , pages 551-560. PMLR.

Bai, Y., Jin, C., and Yu, T. (2020). Near-optimal reinforcement learning with self-play. Advances in neural information processing systems , 33:2159-2170.

Bistritz, I., Zhou, Z., Chen, X., Bambos, N., and Blanchet, J. (2019). Online exp3 learning in adversarial bandits with delayed feedback. Advances in neural information processing systems , 32.

Cesa-Bianchi, N., Dekel, O., and Shamir, O. (2013). Online learning with switching costs and other adaptive adversaries. Advances in Neural Information Processing Systems , 26.

Changuel, N., Sayadi, B., and Kieffer, M. (2012). Online learning for qoe-based video streaming to mobile receivers. In 2012 IEEE Globecom Workshops , pages 1319-1324. IEEE.

Chen, B., Xu, M., Liu, Z., Li, L., and Zhao, D. (2020). Delay-aware multi-agent reinforcement learning for cooperative and competitive environments. arXiv preprint arXiv:2005.05441 .

Chen, Z., Zhou, D., and Gu, Q. (2021). Almost optimal algorithms for two-player markov games with linear function approximation. arXiv preprint arXiv:2102.07404 .

Cui, Q. and Du, S. S. (2022a). Provably efficient offline multi-agent reinforcement learning via strategy-wise bonus. arXiv preprint arXiv:2206.00159 .

Cui, Q. and Du, S. S. (2022b). When is offline two-player zero-sum markov game solvable? arXiv preprint arXiv:2201.03522 .

Daskalakis, C., Golowich, N., and Zhang, K. (2022). The complexity of markov equilibrium in stochastic games. arXiv preprint arXiv:2204.03991 .

Dudik, M., Hsu, D., Kale, S., Karampatziakis, N., Langford, J., Reyzin, L., and Zhang, T. (2011). Efficient optimal learning for contextual bandits. arXiv preprint arXiv:1106.2369 .

Gael, M. A., Vernade, C., Carpentier, A., and Valko, M. (2020). Stochastic bandits with arm-dependent delays. In International Conference on Machine Learning , pages 3348-3356. PMLR.

Gao, M., Xie, T., Du, S. S., and Yang, L. F. (2021). A provably efficient algorithm for linear markov decision process with low switching cost. arXiv preprint arXiv:2101.00494 .

Gao, Z., Han, Y., Ren, Z., and Zhou, Z. (2019). Batched multi-armed bandits problem. Advances in Neural Information Processing Systems , 32.

Gyorgy, A. and Joulani, P. (2021). Adapting to delays and data in adversarial multi-armed bandits. In International Conference on Machine Learning , pages 3988-3997. PMLR.

Howson, B., Pike-Burke, C., and Filippi, S. (2021). Delayed feedback in episodic reinforcement learning. arXiv preprint arXiv:2111.07615 .

Howson, B., Pike-Burke, C., and Filippi, S. (2022). Delayed feedback in generalised linear bandits revisited. arXiv preprint arXiv:2207.10786 .

Huang, B., Lee, J. D., Wang, Z., and Yang, Z. (2021). Towards general function approximation in zero-sum markov games. arXiv preprint arXiv:2107.14702 .

Huang, J., Dai, Y., and Huang, L. (2023). Banker online mirror descent: A universal approach for delayed online bandit learning. arXiv preprint arXiv:2301.10500 .

Ito, S., Hatano, D., Sumita, H., Takemura, K., Fukunaga, T., Kakimura, N., and Kawarabayashi, K.-I. (2020). Delay and cooperation in nonstochastic linear bandits. Advances in Neural Information Processing Systems , 33:4872-4883.

Jin, C., Allen-Zhu, Z., Bubeck, S., and Jordan, M. I. (2018). Is q-learning provably efficient? Advances in neural information processing systems , 31.

Jin, C., Liu, Q., Wang, Y., and Yu, T. (2021). V-learning-a simple, efficient, decentralized algorithm for multiagent rl. arXiv preprint arXiv:2110.14555 .

Jin, C., Liu, Q., and Yu, T. (2022a). The power of exploiter: Provable multi-agent rl in large state spaces. In International Conference on Machine Learning , pages 10251-10279. PMLR.

Jin, T., Lancewicki, T., Luo, H., Mansour, Y., and Rosenberg, A. (2022b). Near-optimal regret for adversarial mdp with delayed bandit feedback. arXiv preprint arXiv:2201.13172 .

Jin, Y., Muthukumar, V., and Sidford, A. (2022c). The complexity of infinite-horizon general-sum stochastic games. arXiv preprint arXiv:2204.04186 .

Joulani, P., Gyorgy, A., and Szepesvári, C. (2013). Online learning under delayed feedback. In International Conference on Machine Learning , pages 1453-1461. PMLR.

Kong, D., Salakhutdinov, R., Wang, R., and Yang, L. F. (2021). Online sub-sampling for reinforcement learning with general function approximation. arXiv preprint arXiv:2106.07203 .

Lancewicki, T., Rosenberg, A., and Mansour, Y. (2022). Learning adversarial markov decision processes with delayed feedback. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7281-7289.

Lancewicki, T., Segal, S., Koren, T., and Mansour, Y. (2021). Stochastic multi-armed bandits with unrestricted delay distributions. In International Conference on Machine Learning , pages 5969-5978. PMLR.

Lattimore, T. and Szepesvári, C. (2020). Bandit algorithms . Cambridge University Press.

Li, G., Chi, Y., Wei, Y ., and Chen, Y . (2022). Minimax-optimal multi-agent rl in markov games with a generative model. In Advances in Neural Information Processing Systems .

Liu, Q., Yu, T., Bai, Y ., and Jin, C. (2021). A sharp analysis of model-based reinforcement learning with self-play. In International Conference on Machine Learning , pages 7001-7010. PMLR.

Mahmood, A. R., Korenkevych, D., Komer, B. J., and Bergstra, J. (2018). Setting up a reinforcement learning task with a real-world robot. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 4635-4640. IEEE.

Mao, W. and Ba¸ sar, T. (2022). Provably efficient reinforcement learning in decentralized general-sum markov games. Dynamic Games and Applications , pages 1-22.

Nash, J. (1951). Non-cooperative games. The Annals of Mathematics , 54(2):286-295.

Perchet, V., Rigollet, P., Chassang, S., and Snowberg, E. (2016). Batched bandit problems. The Annals of Statistics , pages 660-681.

Qiao, D., Yin, M., Min, M., and Wang, Y.-X. (2022). Sample-efficient reinforcement learning with loglog (t) switching cost. arXiv preprint arXiv:2202.06385 .

Quanrud, K. and Khashabi, D. (2015). Online learning with adversarial delays. Advances in neural information processing systems , 28.

Shapley, L. S. (1953). Stochastic games. Proceedings of the national academy of sciences , 39(10):1095-1100.

Song, Z., Mei, S., and Bai, Y. (2021). When can we learn general-sum markov games with a large number of players sample-efficiently? arXiv preprint arXiv:2110.04184 .

Sutton, R. S. and Barto, A. G. (2018). Reinforcement learning: An introduction . MIT press.

Thune, T. S., Cesa-Bianchi, N., and Seldin, Y. (2019). Nonstochastic multiarmed bandits with unrestricted delays. Advances in Neural Information Processing Systems , 32.

Tian, Y., Wang, Y., Yu, T., and Sra, S. (2021). Online learning in unknown markov games. In International conference on machine learning , pages 10279-10288. PMLR.

Vakili, S., Ahmed, D., Bernacchia, A., and Pike-Burke, C. (2023). Delayed feedback in kernel bandits. arXiv preprint arXiv:2302.00392 .

Van Der Hoeven, D. and Cesa-Bianchi, N. (2022). Nonstochastic bandits and experts with armdependent delays. In International Conference on Artificial Intelligence and Statistics . PMLR.

Vernade, C., Cappé, O., and Perchet, V. (2017). Stochastic bandit models for delayed conversions. arXiv preprint arXiv:1706.09186 .

Vernade, C., Carpentier, A., Lattimore, T., Zappella, G., Ermis, B., and Brueckner, M. (2020). Linear bandits with stochastic delayed feedback. In International Conference on Machine Learning , pages 9712-9721. PMLR.

Wainwright, M. J. (2019). High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press.

Wang, T., Zhou, D., and Gu, Q. (2021). Provably efficient reinforcement learning with linear function approximation under adaptivity constraints. Advances in Neural Information Processing Systems , 34:13524-13536.

Xie, Q., Chen, Y., Wang, Z., and Yang, Z. (2020). Learning zero-sum simultaneous-move markov games using function approximation and correlated equilibrium. In Conference on learning theory , pages 3674-3682. PMLR.

Xiong, W., Zhong, H., Shi, C., Shen, C., Wang, L., and Zhang, T. (2022a). Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game. arXiv preprint arXiv:2205.15512 .

Xiong, W., Zhong, H., Shi, C., Shen, C., and Zhang, T. (2022b). A self-play posterior sampling algorithm for zero-sum Markov games. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 24496-24523. PMLR.

Yan, Y., Li, G., Chen, Y ., and Fan, J. (2022). Model-based reinforcement learning is minimax-optimal for offline zero-sum markov games. arXiv preprint arXiv:2206.04044 .

Zhang, K., Kakade, S., Basar, T., and Yang, L. (2020). Model-based multi-agent rl in zero-sum markov games with near-optimal sample complexity. Advances in Neural Information Processing Systems , 33:1166-1178.

Zhang, K., Yang, Z., and Ba¸ sar, T. (2021a). Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control , pages 321-384.

Zhang, Y., Zhang, R., Li, G., Gu, Y., and Li, N. (2022a). Multi-agent reinforcement learning with reward delays. arXiv preprint arXiv:2212.01441 .

Zhang, Z., Ji, X., and Du, S. (2021b). Is reinforcement learning more difficult than bandits? a near-optimal algorithm escaping the curse of horizon. In Conference on Learning Theory , pages 4528-4531. PMLR.

Zhang, Z., Jiang, Y., Zhou, Y., and Ji, X. (2022b). Near-optimal regret bounds for multi-batch reinforcement learning. arXiv preprint arXiv:2210.08238 .

Zhong, H., Xiong, W., Tan, J., Wang, L., Zhang, T., Wang, Z., and Yang, Z. (2022). Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets. arXiv preprint arXiv:2202.07511 .

Zhong, H., Yang, Z., Wang, Z., and Jordan, M. I. (2021). Can reinforcement learning find stackelberg-nash equilibria in general-sum markov games with myopic followers? arXiv preprint arXiv:2112.13521 .

Zimmert, J. and Seldin, Y. (2020). An optimal algorithm for adversarial bandits with arbitrary delays. In International Conference on Artificial Intelligence and Statistics , pages 3285-3294. PMLR.

A Proof of Theorem 1

We first prove (7). First, since the delays are i.i.d. subexponential random variables, by standard concentration inequality and uniform bound Wainwright (2019), we have the following lemma:

Lemma 5. Let { τ t } ∞ t =1 be i.i.d ( v, b ) subexponential random variables with expectation E [ τ ] . Then we have

Define the good event G = {∃ t ≥ 1 : τ t ≥ E [ τ ] + C τ } . By Lemma 5, the good event happens with probability at least 1 -δ . Therefore, conditioned on the good event happens, in each batch of the algorithm, we only need to run E [ τ ] + C τ episodes and the algorithm can observe enough feedback to finish this batch.

The regret in a batch can be divided into two parts: the first part is the regret caused by running the algorithm in the undelayed environment; the second part is caused by running extra episodes to wait for delayed feedback. The first part can be simply bounded by ˜ Regret( K ) . For the second part, since each episode contributes at most H regret, the total regret can be bounded by

For the second inequality, we have the following lemma which reveals the relation between the number of observed feedback and the actual number of episodes using quantile function.

Lemma 6. Denote the number of observed feedback at time t as n t . For any quantile q and any t , we have

Proof. Define I { τ s ≤ d τ ( q ) } to be an indicator that on time s that the delay is smaller than d τ ( q ) . Note that E [ I { τ s ≤ d τ ( q ) } ] ≥ q . Thus,

where the third inequality follows from the relative Chernoff bound, and the last inequality is since ∑ t s =1 E [ I { τ s ≤ d τ ( q ) } ] ≥ q · t .

Lemma 6 means, if the length of the batch in the undelayed environment m> log( K/δ ) q , then with probability at least 1 -δ K , if we run the batch for m q + d τ ( q ) episodes, we will observe at least m feedbacks, which is enough for the algorithm to finish this batch. On the other hand, if m ≤ log( K/δ ) q , then the regret of this batch can be trivially bounded by H log( K/δ ) q . Hence the total regret can be bounded by

which finishes the proof.

B Details of Algorithms for Linear Bandits

The phase elimination algorithm for linear bandits in Lattimore and Szepesvári (2020) is as follows. This algorithm is naturally a multi-batched algorithm, with each phase as one batch. Since the length of phase ℓ is O (2 ℓ ) , the number of phases is O (log K ) . Note that in each phase the policy is pre-calculated at the beginning of the phase, and does not rely on the data of current phase.

Algorithm 3 Phase Elimination

1: Input: A ⊂ R d , β > 1 , δ

2: Set ℓ = 1 and let A 1 = A

3: Let t ℓ = t be the current timestep and find G -optimal design π ℓ ∈ P ( A ℓ ) with Supp( π ℓ ) ≤ d ( d +1) / 2 that maximises

4: Let ε ℓ = 2 -ℓ and

5: Choose each action a ∈ A ℓ exactly T ℓ ( a ) times.

6: Calculate the empirical estimate:

using observable data in this phase.

7: Eliminating low reward arms:

8: ℓ ← ℓ +1 and go to Line 2.

Theorem 7 (Theorem 22.1 in Lattimore and Szepesvári (2020)) . With probability at least 1 -δ , the regret of Algorithm 5 satisfies

where C > 0 is a universal constant. If δ = O (1 /K ) , then E [ Regret( K ) ] ≤ C √ Kd log( AK ) for an appropriately chosen universal constant C > 0 .

For infinite action set A = [ -1 , 1] d , we use a covering technique: we find a ϵ -covering of the action space, and run Phase Elimination on the covering set. The covering number of the set [ -1 , 1] d is O ( d log(1 /ϵ )) . And the error caused by using ϵ -covering will lead to a O ( ϵK ) increase in regret. Therefore, by setting ϵ = 1 K , the regret of the whole algorithm is O ( d √ K log K ) .

C Results for Bandits and MDPs

We now apply Theorem 1 to different bandit and MDP problems.

In multi-armed bandit settings, using the BaSE algorithm Gao et al. (2019), we can get the following bound.

Corollary 4. For multi-armed bandit with delayed feedback, running Algorithm 2 with ALG being the BaSE algorithm in Gao et al. (2019), we can upper bound the regret by

̸

for any q ∈ (0 , 1) , where ∆ i is the optimal gap of the i th arm. In addition, if the delays satisfy Assumption 2, the regret can be upper bounded by

̸

This result recovers the SOTA result in Lancewicki et al. (2021) up to a log K factor in the delaydependent term.

In linear bandit settings, using the Phase Elimination algorithm (Lattimore and Szepesvári, 2020), we can get the following bound.

Corollary 5. For linear bandit with delayed feedback, let d be the dimension of the feature space. When the arm set A is finite, running Algorithm 2 with ALG being the Phase Elimination algorithm in Lattimore and Szepesvári (2020), we can upper bound the regret by

for any q ∈ (0 , 1) . In addition, if the delays satisfy Assumption 2, the regret can be upper bounded by

When the action set is infinitely large (for example, A = [ -1 , 1] d ), we can use techniques such as covering to obtain regret bounds whose main term scales as ˜ O ( d √ K ) . Compared with the result in Howson et al. (2021) which scales as ˜ O ( d √ K + d 3 / 2 E [ τ ]) , our bound is tighter in the delay-dependent term as we remove the d 3 / 2 factor.

In tabular MDP settings, using the policy elimination algorithm Zhang et al. (2022b), we can get the following bound.

Corollary 6. For tabular MDP with delayed feedback, running Algorithm 2 with ALG being the policy elimination algorithm in Zhang et al. (2022b), we can upper bound the regret by

for any q ∈ (0 , 1) , where ι is some log factor. In addition, if the delays satisfy Assumption 2, the regret can be upper bounded by

Compared with the previous result √ H 3 SAKι + H 2 SA E [ τ ] log K in Howson et al. (2022), our bound removes the SA factor in the delay-dependent term, therefore is tighter.

We can also apply our framework to linear MDP and MDP with general function approximation settings with delay. These settings have not been studied before, therefore our result is novel. In linear MDP settings, using the algorithm in Gao et al. (2021) we can get the following bound.

Corollary 7. For linear MDP with delayed feedback, let d be the dimension of the feature space. Running Algorithm 2 with ALG being the algorithm in Gao et al. (2021), we can upper bound the regret by

for any q ∈ (0 , 1) . In addition, if the delays satisfy Assumption 2, the regret can be upper bounded by

And in MDP with general function approximation, using the algorithm in Kong et al. (2021) we get the following bound.

Corollary 8. For MDP with general function approximation and delayed feedback, let dim E ( F , ϵ ) be the ϵ -eluder dimension of the function class F . Running Algorithm 2 with ALG being the algorithm in Kong et al. (2021), we can upper bound the regret by

for any q ∈ (0 , 1) . In addition, if the delays satisfy Assumption 2, the regret can be upper bounded by

D Proofs for Tabular Markov Games

D.1 Multi-batched Optimistic Nash-VI

In this part, we list some of the important lemmas for the tabular Markov game algorithm, most of which are proven in the previous literature Liu et al. (2021). These lemmas are very useful for proving the regret bound in our main theorem.

We denote V k , Q k , π k , µ k and ν k for values and policies at the beginning of the k -th episode. In particular, N k h ( s, a, b ) is the number we have visited the state-action tuple ( s, a, b ) at the h -th step

before the k -th episode. N k h ( s, a, b, s ′ ) is defined by the same token. Using this notation, we can further define the empirical transition by ̂ P k h ( s ′ | s, a, b ) := N k h ( s, a, b, s ′ ) /N k h ( s, a, b ) . If N k h ( s, a, b ) = 0 , we set ̂ P k h ( s ′ | s, a, b ) = 1 /S .

We define the empirical variance operator

The bonus terms can be written as

and true variance operator where

and the lower-order term

for some absolute constant C > 0 .

The following lemma is the standard concentration result.

Lemma 8 (Lemma 21, Liu et al. (2021)) . Let c 1 be some large absolute constant. Define event E 1 to be: for all h, s, a, b, s ′ and k ∈ [ K ] ,

We have P ( E 1 ) ≥ 1 -p .

The following lemma shows the upper and lower bounds are indeed the upper and lower bounds of the best responses.

Lemma 9 (Lemma 22, Liu et al. (2021)) . Suppose event E 1 holds. Then for all h, s, a, b and k ∈ [ K ] , we have

Lemma 10 (Lemma 23, Liu et al. (2021)) . Suppose event E 1 holds. Then for all h, s, a, b and k ∈ [ K ] , we have

Proof of Theorem 4. Suppose event E 1 happens. We define

Using standard decomposition techniques we have

where c 4 is some absolute constant. Define c 5 := 4 c 2 c 4 C + c 3 and κ := 1 + c 5 /H . Now we have

Recursing this argument for h ∈ [ H ] and summing over k ,

It is easy to check that { ζ k h } and { ξ k h } are martingales. By Azuma-Hoeffding inequality, with probability at least 1 -p ,

Now we deal with the rest terms. Note that under the doubling epoch update framework, despite those episodes in which an update is triggered, the number of visits of ( s, a ) between the i -th update of ˆ P s,a and the i +1 -th update of ˆ P s,a do not exceed 2 i -1 .

We define i max = max { i | 2 i -1 ≤ KH } = ⌊ log 2 ( KH ) ⌋ +1 . To facilitate the analysis, we first derive a general deterministic result. Let K be the set of indexes of episodes in which no update is triggered. By the update rule we have ∣ ∣ K C ∣ ∣ ≤ SA ( log 2 ( KH ) + 1 ) . Let h 0 ( k ) be the first time an update is triggered in the k -th episode if there is an update in this episode and otherwise H +1 . Define X = { ( k, h ) | k ∈ K C , h 0 ( k ) + 1 ≤ h ≤ H } .

Let w = { w k h ≥ 0 | 1 ≤ h ≤ H, 1 ≤ k ≤ K } be a group of non-negative weights such that w k h ≤ 1 for any ( k, h ) ∈ [ H ] × [ K ] and w k h = 0 for any ( k, h ) ∈ X . As in Zhang et al. (2021b), We can calculate

By plugging w k h = V h V π k h +1 ( s k h , a k h , b k h ) I [( k, h ) / ∈ X ] and w k h = I [( k, h ) / ∈ X ] we have

By the Law of total variation and standard martingale concentration (see Lemma C.5 in Jin et al. (2018) for a formal proof), with probability at least 1 -p , we have

By Pigeon-hole argument,

Putting all relations together, we obtain that

which completes the proof.

D.2 Multi-batched Algorithms for Linear Markov Games

Algorithm 5 Multi-batched Algorithm for Linear Markov Games

Proof of Theorem 3. Let { k 1 , · · · , k N batch } be the episodes where the learner updates the policies. Moreover, we define b k = max { k i : i ∈ [ N batch ] , k i ≤ k } . Moreover, we denote the marginals of π k h as µ k h and ν k h .

Part 1: Upper bound of the batch number. By the updating rule, we know there exists one h ∈ [ H ] such that det(Λ k i h ) > η · det(Λ k i -1 h ) , which further implies that

This yields that

On the other hand, for any h ∈ [ H ] , we have

Furthermore, by the fact that ∥ ϕ ( · , · ) ∥ 2 ≤ 1 , we have

which implies that

Combining (34), (35), and (36), we obtain that

which concludes the first part of the proof.

Part2: Regret. We need the following two lemmas to bound the regret.

Lemma 11. For any ( s, a, b, k, h ) , it holds that

Proof. See Lemma 5 of Xie et al. (2020) for a detailed proof.

Lemma 12. For all ( s, a, b, h, k ) and any fixed policy pair ( µ, ν ) , it holds with probability at least 1 -δ that

Proof. See Lemma 3 of Xie et al. (2020) for a detailed proof.

We define

By Lemma 11, we have which implies that

where the last inequality uses the elliptical potential lemma (Abbasi-Yadkori et al., 2011). Combining (37), (40), and (41), together with the fact that η = 2 , we obtain

where the second inequality uses that ϵ = 1 / ( KH ) . By definition, we have

Meanwhile, for any ( s, a, b, k, h ) , we have

By Lemma 12, we further have

Plugging (39) into (38) gives that

Hence, we have

where the last inequality uses the Azuma-Hoeffding inequality. Note that

which concludes the proof.

D.3 Multi-batched V-learning

In this section, we introduce the multi-batched version of V-learning (Jin et al., 2021) for general-sum Markov games. Our goal is to minimize the following notion of regret. Let V π i, 1 ( s 1 ) be the expected cumulative reward that the i th player will receive if the game starts at initial state s 1 at the 1 st step and all players follow joint policy π . Let ˆ π k be the policy executed at the k th episode. The regret is defined as

where V † , ˆ π k -j,h j,h is the best response of the j th player when other players follow ˆ π .

Algorithm 6 Multi-batched V-learning

1: Initialize: V h ( s ) ← H + 1 -h, N h ( s ) ← 0 , π h ( a | s ) ← 1 /A , trigger set L ← { 2 i -1 | 2 i ≤ KH,i = 1 , 2 , . . . } 2: for episode k = 1 , ..., K do 3: Receive s 1 . 4: for h = 1 , ..., H do 5: take action a h ∼ π ˜ k h ( · | s h ) , observe r h and s h +1 . 6: t = N h ( s h ) ← N h ( s h ) + 1 . 7: ˜ V h ( s h ) ← (1 -α t ) ˜ V h ( s h ) + α t ( r h + V h +1 ( s h +1 ) + β ( t ) ) 8: V h ( s h ) ← min { H +1 -h, ˜ V h ( s h ) } 9: π k h ( · | s h ) ← ADV_BANDIT_UPDATE ( a h , H -r h -V h +1 ( s h +1 ) H ) on ( s h , h ) th adversarial bandit. 10: end for 11: if ∃ s , N h ( s ) ∈ L then 12: ˜ k ← k 13: end if 14: end for

The multi-batched V-learning algorithm maintains a value estimator V h ( s ) , a counter N h ( s ) , and a policy π h ( · | s ) for each s and h . We also maintain S × H different adversarial bandit algorithms. At step h in episode k , the algorithm is divided into three steps: policy execution, V -value update, and policy update. In policy execution step, the algorithm takes action a h according to π ˜ k h , and observes reward r h and the next state s h +1 , and updates the counter N h ( s h ) . Note that π ˜ k h is updated only when the visiting count of some state N h ( s ) doubles. Therefore it ensures a low batch number.

In the V -value update step, we update the estimated value function by

where the learning rate is defined as

and β ( t ) is the bonus to promote optimism.

In the policy update step, the algorithm feeds the action a h and its "loss" H -r h + V h +1 ( s h +1 ) H to the ( s h , h ) th adversarial bandit algorithm. Then it receives the updated policy π h ( · | s h ) .

The ADV_BANDIT_UPDATE subroutine Consider a multi-armed bandit problem with adversarial loss, where we denote the action set by B with |B| = B . At round t , the learner determines a policy θ t ∈ ∆ B , and the adversary chooses a loss vector ℓ t ∈ [0 , 1] B . Then the learner takes

an action b t according to policy θ t , and receives a noisy bandit feedback ˜ ℓ t ( b t ) ∈ [0 , 1] , where E [ ˜ ℓ t ( b t ) | ℓ t , b t ] = ℓ t ( b t ) . Then, the adversarial bandit algorithm performs updates based on b t and ˜ ℓ t ( b t ) , and outputs the policy for the next round θ t +1 .

We first state our requirement for the adversarial bandit algorithm used in V-learning, which is to have a high probability weighted external regret guarantee as follows.

Assumption 4. For any t ∈ N and any δ ∈ (0 , 1) , with probability at least 1 -δ , we have

We further assume the existence of an upper bound Ξ( B,t, log(1 /δ )) ≥ ∑ t t ′ =1 ξ ( B,t ′ , log(1 /δ ) ) where (i) ξ ( B,t, log(1 /δ )) is non-decreasing in B for any t, δ ; (ii) Ξ( B,t, log(1 /δ )) is concave in t for any B,δ .

As is shown in Jin et al. (2021), the Follow-the-Regularized-Leader (FTRL) algorithm satisfies the assumption with bounds ξ ( B,t, log(1 /δ )) ≤ O ( √ HB log( B/δ ) /t ) and Ξ( B,t, log(1 /δ )) ≤ O ( √ HBt log( B/δ )) . And we will use FTRL as the subroutine in our algorithm.

For two-player general sum Markov games, we let the two players run Algorithm 6 independently. Each player will use her own set of bonus that depends on the number of her actions. We have the following theorem for the regret of multi-batched V-learning.

Theorem 13. Suppose we choose subroutine ADV_BANDIT_UPDATE as FTRL. For any δ ∈ (0 , 1) and K ∈ N , let ι = log( HSAK/δ ) . Choose learning rate α t and bonus { β ( t ) } K t =1 as β ( t ) = c · √ H 3 Aι/t so that ∑ t i =1 α i t β ( i ) = Θ ( √ H 3 Aι/t ) for any t ∈ [ K ] , where A = max i A i Then, with probability at least 1 -δ , after running Algorithm for K episodes, we have

And the batch number is O ( HS log K ) .

Proof. The proof mainly follows Jin et al. (2021). Below we list some useful lemmas, which have already been proved in Jin et al. (2021).

Lemma 14 (Lemma 10, Jin et al. (2021)) . The following properties hold for α i t :

1 √ t ≤ ∑ t i =1 α i t √ i ≤ 2 √ t and 1 t ≤ ∑ t i =1 α i t i ≤ 2 t for every t ≥ 1 .

max i ∈ [ t ] α i t ≤ 2 H t and ∑ t i =1 ( α i t ) 2 ≤ 2 H t for every t ≥ 1 .

∑ ∞ t = i α i t = 1 + 1 H for every i ≥ 1 .

The next lemma shows that V k h is an optimistic estimation of V † , ˆ π k -j,h j,h

Lemma 15 (Lemma 13, Jin et al. (2021)) . For any δ ∈ (0 , 1] , with probability at least 1 -δ , for any ( s, h, k, j ) ∈ S × [ H ] × [ K ] × [ m ] , V k ( s ) ≥ V † ,ν k ( s )

And we define the pessimistic pessimistic V-estimations V that are defined similarly as ˜ V and V . Formally, let t = N k h ( s ) , and suppose s was previously visited at episodes k 1 , . . . , k t < k at the h -th step. Then

The next lemma shows V k j,h ( s ) is a lower bound of V ˆ π k h j,h ( s ) .

Lemma 16 (Lemma 14, Jin et al. (2021)) . For any δ ∈ (0 , 1] , with probability at least 1 -δ , the following holds for any ( s, h, k, j ) ∈ S × [ H ] × [ K ] × [ m ] and any player j, V k j,h ( s ) ≤ V ˆ π k h j,h ( s ) .

It remains to bound the gap ∑ K k =1 max j ( V k 1 ,j -V k 1 ,j ) ( s 1 ) . For each player j we define δ k j,h := V k j,h ( s k h ) -V k j,h ( s k h ) ≥ 0 . The non-negativity here is a simple consequence of the update rule and induction. We need to bound δ k h := max j δ k j,h . Let n k h = N k h ( s k h ) and suppose s k h was previously visited at episodes k 1 , . . . , k n k h < k at the h -th step. Now by the update rule,

The third line holds because we use FTRL as the subroutine, β j ( t ) = c · √ H 3 A j ι/t , and due to our doubling scheme we have β ( n ˜ k h ) ≤ 2 β ( n k h ) . And in the last step we have used ∑ t i =1 α i t β j,i = Θ ( Hξ ( A j , t, ι ) + √ H 3 Aι/t ) .

The remaining step is the same as Jin et al. (2021). Summing the first two terms w.r.t. k ,

Putting them together,

Recursing this argument for h ∈ [ H ] gives

By pigeonhole argument,

where in the last step we have used concavity. Finally take the sum w.r.t h ∈ [ H ] , and plugging Ξ( B,t, log(1 /δ )) ≤ O ( √ HBt log( B/δ )) we have

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of sequential decision-making problems with delayed feedback. The authors propose a reduction from standard sequential decision-making algorithms to the following problem providing regret guarantees in many settings: linear bandits, tabular RL, linear and general function approximations RL, tabular and linear zero-sum Markov Games, tabular general-sum Markov Games.\n\nSTRENGTHS:\n- The authors improve previous results in the delayed feedback setting for multi-armed bandit, linear bandit, tabular MDP, tabular multi-player general sum MGs.\n- The authors provide the first regret guarantees (in the delayed feedback setting) for linear MDPs, general-function approximation MDPs, and tabular and linear two-player zero-sum MGs.\n- The strategy is simple: \n   - single-agent: apply multi-batched algorithm for RL to the delayed feedback setting running some extra steps depending on the feedback delay $\\tau_k$.\n  - zero-sum tabular and general-sum tabular: using the doubling trick to update the policy.\n  - zero-sum linear: update the policy when there exists $h \\in [H]$ such that $\\text{det}(\\Lambda^k_h) > \\eta \\cdot \\text{det}(\\Lambda_h)$.\n\nWEAKNESSES:\n### Presentation\n\nThe presentation of the paper can be improved. The paper proposes a general strategy to convert multi-batch algorithms to the delayed feedback setting, changing (if I understood correctly) when the policy is updated. This is a simple and elegant transformation and it is not very clear from the presentation where the authors divide the transformations into three sections without providing this connection between them. I suggest providing an algorithm for the general setting which changes the update depending on the setting. \nMoreover, there are some minor issues in the way the algorithms are presented:\n- provide the input parameters in a clear way (e.g. algorithm 1 what is $l_m$?, algorithm 2 how can we compute $\\tau_t$ since it is a random variable, algorithm 4 what is i?)\n- provide output to the algorithms.\n\n\n### Novelty\n\nThe simplicity of the strategy can be seen as a lack of novelty (due also to the previous point (see Presentation)). I suggest making more clear the challenges.\n\nQUESTIONS:\nSee weaknesses. \n\nSome minor questions:\n- In algorithm 2, do we know $\\tau_t$ or are we waiting until we do not receive the feedback, and then we can evaluate $\\tau_t$?\n- From a technical point of view what are the main challenges in converting regret bounds from the \"classic\" setting to the delayed feedback one?\n\nLIMITATIONS:\nThe paper is mainly theoretical and the algorithms presented are not easily usable in practice.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission studies the problem of learning from delayed feedback in various online learning settings including multi-armed bandits, linear bandits, linear MDPs, RL with function approximation, and various markov game settings (among others). Delays are assumed to be i.i.d. random variables. Additionally, they are sometimes assumed to be subexponential random variables. \n\nThe authors main result is a framework for sequential decision making with delayed feedback. Their framework makes use of \"multi-batched\" algorithms, which, as the name suggests, runs in batches. In each batch, a multi-batched algorithm outputs a sequence of policies to be used one-after-one (which are trained using data from previous batches), as well as a stopping criterion which decides when to stop the current batch. After a batch is run, the new data is incorporated with the existing data and is used to produce a new sequence of policies and stopping rule for the next round.\n\nAt a high level, the framework converts a multi-batched algorithm for the setting of interest (e.g. linear bandits) to one which can handle delayed feedback. The key idea is to run the policy given by the multi-batched algorithm for some extra time-steps in order to satisfy the stopping criteria. The authors are then able to bound the additional regret due to the delayed feedback by bounding the number of additional steps needed to satisfy the stopping criteria. \n\nBy instantiating their framework in the various settings listed above, they can obtain provable guarantees on the learner's regret. The rates obtained improve upon existing results in some settings (e.g. linear bandits and tabular MDPs), are slightly worse in others (e.g. multi-armed bandits), and are the first of their kind in other settings (e.g., RL with general function approximation).\n\nSTRENGTHS:\nThe problem of delayed feedback in online learning settings is a well-motivated problem with many real-world applications. Furthermore, the authors are the first to obtain results for learning with delayed feedback in a variety of (important) settings such as RL with general function approximation, as well as improve upon existing results in others such as linear bandits. \n\nThe two main strengths of this paper are the simplicity/clarity of the proposed framework and the sheer number of results the authors are able to obtain using their framework. It is impressive that results for so many important settings are able to be obtained by applying one simple framework. Moreover, all that is required to use their framework is a multi-batched algorithm, which already exist in the literature for many online learning settings.\n\nWEAKNESSES:\nAt times, the writing feels a bit rushed and as a result the submission may be a hard read for someone who is not an expert in RL theory. Specifically, I felt that the mutli-agent results were somewhat lacking in detail. (For example, why is the CCE the \"right\" solution concept, what happens when all players do/don't play a CCE policy, etc.) This is probably due to the fact that the authors have lots of results, which they wish to highlight in the main body of the paper. One suggestion would be to move either subsection 5.1, 5.2, or 5.3 to the Appendix, and use the extra space to provide more background/detail/hand-holding for the reader.\n\nQUESTIONS:\nI was a bit confused by the notation in line 9 of Algorithm 2. Shouldn't this read (both in words and mathematically) \"Collect trajectory feedback in this batch that is observed **by** the end of the episode\"?\n\nWould your framework be able to obtain stronger results if, instead of Assumption 2, the delays are assumed to be subGaussian random variables?\n\nA reference is missing to \"Banker Online Mirror Descent: A Universal Approach for Delayed Online Bandit Learning\" by Huang et al.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new framework for handling stochastic delayed feedback in general sequential decision-making problems, encompassing bandits, single-agent Markov decision processes (MDPs), and Markov games (MGs). The authors introduce a novel reduction-based framework that converts any multi-batched algorithm for sequential decision-making with instantaneous feedback into a sample-efficient algorithm capable of managing stochastic delays. They provide various examples, demonstrating the efficacy of their framework in different scenarios, and present several new results, improving existing findings for linear bandits, tabular RL, and tabular MGs.\n\nSTRENGTHS:\nThe paper addresses a significant and practical issue in sequential decision-making: stochastic delayed feedback. It is well-motivated with practical examples like recommendation systems, robotics, and video streaming.\nThe paper presents a novel, reduction-based framework that shows versatility across multiple domains (bandits, single-agent MDPs, and MGs), exhibiting a comprehensive approach.\nThe proposed solution offers theoretical advancements by improving the regret bounds for linear bandits and tabular RL and provides the first theoretical guarantees in RL with function approximation and multi-agent RL settings.\nThe authors effectively integrate existing multi-batched algorithms into their framework, illustrating the generality of their approach.\n\nWEAKNESSES:\nIt seems that the contribution is rather incremental. The idea that multi-batched algorithms can be used in delayed setting is not new. What appears to be new is some new theorems bounding the regret when using multi-batched algorithms for delayed settings in a variety of settings. \nThe authors could strengthen their paper with some simple experiments to add to the theory\n\nQUESTIONS:\nCan you elaborate more on the specific limitations of the current state-of-the-art methods that your framework is intended to overcome?\nCan your framework be generalized to handle other types of delays, or is it specifically designed for stochastic delays?\n\nLIMITATIONS:\nCan we develop new algorithms that work even better than existing multi-batch algorithms? \nThe paper leaves open the question of whether their results are tight for MDP and MG with delayed feedback. Thus, there is room for future work to tighten these results.\nThe paper suggests that a multi-batched algorithm with a smaller batch number for tabular Markov games could be derived, which may provide more efficiency, but does not provide it in this paper.\nIt remains unclear if their findings and the proposed framework will translate effectively into practical scenarios. The authors have not provided any real-world evaluations or use-cases to support their claims.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a general framework for analysing any ‘mini-batch’ or rarely switching algorithm in the presence of delayed feedback. This approach covers bandits, finite horizon MDPs and finite horizon Markov games. The results provided match or improve on the best-known results for delayed feedback in settings that have been studied before, and provide the first bounds in some settings where delayed feedback has not been studied before.\n\nSTRENGTHS:\nIt is nice to have a comprehensive study of delayed feedback in various settings. It is interesting that phase-based algorithms allow one to deal with delayed feedback effectively in a variety of settings. It is nice that the general framework provided in this paper allows us to recover the same bounds as for algorithms for specific settings and provides new bounds in settings where delayed feedback has not been studied previously.\n\nWEAKNESSES:\nThe writing and clarity could be improved. I also found the structure quite confusing with a couple of pages dedicated to Markov games while a lot of the content needed to understand that section (e.g. related work and algorithms) relegated to the appendix. The amount of space dedicated to Markov games also meant that there was not sufficient room to discuss the results for delayed feedback in bandits/RL, nor for all details of the method/results to be provided (see the many questions below).\nThe idea of using phase-based algorithms which switch arms less frequently for delayed feedback is not new and has been used in e.g. Lancewicki et al (2021), Howson et al (2021), Pike-Burke et al (2018), Vakili et al (2023),… This should be stated in section 4. Additionally, I believe Vakili et al (2023) also provide the same bound for the linear bandit setting with delayed feedback as is established in this paper, so this should also be checked and added. \n\nReferences: Vakili, Sattar, et al. \"Delayed Feedback in Kernel Bandits.\" ICML, 2023.\nPike-Burke, Ciara, et al. \"Bandits with delayed, aggregated anonymous feedback.\" ICML, 2018\n\nQUESTIONS:\nThe two results in e.g. theorem 1 are a bit confusing and the difference between them is not well explained. Is the second one (7) on the expected regret and the first (6) on the high probability regret? For (6), I am assuming that q is a parameter which can be chosen, can any guidance be provided for how to choose it? Does it depend on \\delta?\nI am interested in whether experimentally these rarely switching algorithms actually perform better than algorithms tailored to the specific setting, or just versions of optimistic algorithms that only use available data?\nA couple of times ‘minimax optimal’/’tight’ is stated. Are there lower bounds for all these settings with delayed feedback?\nCan the lower bounds/best results for the non-delayed settings be added to the table of results to enable us to clearly understand what the impact of delayed feedback is? \nIn the general algorithm provided in section 4 and algorithm 2, do we need to know the delay distribution to define the batch lengths? The example in the text would suggest that we do, but the pseudo-code does not mention it.\nIn the introduction it is mentioned that the sub-exponential assumption on the delay distribution is only sometimes needed, yet I cannot find any discussion of when it is used for the main results. Is it needed for all results in Theorem 1?\n\nLIMITATIONS:\nVery brief discussion of limitations in conclusion", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}