Title: A Best-of-both-worlds Algorithm for Bandits with Delayed Feedback with Robustness to Excessive Delays
Venue: NeurIPS 2024
Accepted: yes
Keywords: Best-of-both-worlds

delayed bandit feedback
Abstract: We propose a new best-of-both-worlds algorithm for bandits with variably delayed feedback. In contrast to prior work, which required prior knowledge of the maximal delay $d_{\max}$ and had a linear dependence of the regret on it, our algorithm can tolerate arbitrary excessive delays up to order $T$ (where $T$ is the time horizon). The algorithm is based on three technical innovations, which may all be of independent interest: (1) We introduce the first implicit exploration scheme that works in best-of-both-worlds setting. (2) We introduce the first control of distribution drift that does not rely on boundedness of delays. The control is based on the implicit exploration scheme and adaptive skipping of observations with excessive delays. (3) We introduce a procedure relating standard regret with drifted regret that does not rely on boundedness of delays. At the conceptual level, we demonstrate that complexity of best-of-both-worlds bandits with delayed feedback is characterized by the amount of information missing at the time of decision making (measured by the number of outstanding observations) rather than the time that the information is missing (measured by the delays).

A Best-of-both-worlds Algorithm for Bandits with Delayed Feedback with Robustness to Excessive Delays

Abstract

We propose a new best-of-both-worlds algorithm for bandits with variably delayed feedback. In contrast to prior work, which required prior knowledge of the maximal delay d max and had a linear dependence of the regret on it, our algorithm can tolerate arbitrary excessive delays up to order T (where T is the time horizon). The algorithm is based on three technical innovations, which may all be of independent interest: (1) We introduce the first implicit exploration scheme that works in bestof-both-worlds setting. (2) We introduce the first control of distribution drift that does not rely on boundedness of delays. The control is based on the implicit exploration scheme and adaptive skipping of observations with excessive delays. (3) We introduce a procedure relating standard regret with drifted regret that does not rely on boundedness of delays. At the conceptual level, we demonstrate that complexity of best-of-both-worlds bandits with delayed feedback is characterized by the amount of information missing at the time of decision making (measured by the number of outstanding observations) rather than the time that the information is missing (measured by the delays).

1 Introduction

Delayed feedback is an ubiquitous challenge in real-world applications. Study of multi-armed bandits with delayed feedback has started at least four decades ago in the context of adaptive clinical trials (Simon, 1977, Eick, 1988), the same problem that has earlier motivated introduction of the bandit model itself (Thompson, 1933). We focus on robustness to delay outliers and to the loss generation mechanism. In practice occasional delay outliers are common (e.g., observations that never arrive). Robustness to the loss generation mechanism implies that the algorithm does not need to know whether the losses are stochastic or adversarial, but still provides regret bounds that match the optimal stochastic rates if the losses happen to be stochastic, while guaranteeing the adversarial rates if they are not (so-called best-of-both-worlds regret bounds). Such algorithms are important from a practical viewpoint, because the loss generation mechanism can rarely assumed to be stochastic, but it is still desirable to have tighter regret bounds if it happens to be. From the theoretical perspective both forms of robustness are interesting and challenging, requiring novel analysis tools and yielding better understanding of the problems.

Joulani et al. (2013) have studied multi-armed bandits with delayed feedback under the assumption that the rewards are stochastic and the delays are sampled from a fixed distribution.

̸

̸

They provided a modification of the UCB1 algorithm for stochastic bandits with non-delayed feedback (Auer et al., 2002). They have shown that the regret of the modified algorithm is O ( ∑ i :∆ i > 0 ( log T ∆ i + σ max ∆ i )) , where i indexes the arms, ∆ i is the suboptimality gap of arm i , T is the time horizon (unknown to the algorithm), and σ max is the maximal number of outstanding observations. (An observation is counted as outstanding at round t if it originates from round t or earlier, but due to delay it was not revealed to the algorithm by the end of round t . The number of outstanding observations σ t at round t is the number of actions that have already been played, but their outcome was not observed yet. We also call σ t the [running] count of outstanding observations. The maximal number of outstanding observations σ max is the maximal value that σ t takes and is unknown to the algorithm.) The result implies that in the stochastic setting the delays introduce an additive term in the regret bound, proportional to the maximal number of outstanding observation.

In the adversarial setting, multi-armed bandits with delayed feedback were first analyzed under the assumption of uniform delays (Neu et al., 2010, 2014). For this setting Cesa-Bianchi et al. (2019) have shown an Ω( √ KT + √ dT log K ) lower bound and an almost matching upper bound, where K is the number of arms and d is a fixed delay. The algorithm of Cesa-Bianchi et al. is a modification of the EXP3 algorithm of Auer et al. (2002b). Cesa-Bianchi et al. used a fixed learning rate that is tuned based on the knowledge of d . The analysis is based on control of the drift of the distribution over arms played by the algorithm from round t to round t + d . Thune et al. (2019) and Bistritz et al. (2019) provided algorithms for variable adversarial delays, but under the assumption that the delays are known 'at action time', meaning that the delay d t is known at time t , when the action is taken, rather that at time t + d t , when the observation arrives. The advanced knowledge of delays was used to tune the learning rate and control the drift of played distribution from round t , when an action is played, to round t + d t , when the observation arrives. Alternatively, an advance knowledge of the cumulative delay up to the end of the game could be used for the same purpose. Finally, Zimmert and Seldin (2020) derived an algorithm for the adversarial setting that required no advance knowledge of delays and matched the lower bound of Cesa-Bianchi et al. (2019) within constants. The algorithm and analysis of Zimmert and Seldin avoid explicit control of the distribution drift and are parameterized

by running counts of the number of outstanding observations σ t , which is an empirical quantity that is observed at time t ('at the time of action').

Masoudian et al. (2022) attempted to extend the algorithm of Zimmert and Seldin (2020) to the best-of-both-worlds setting. The stochastic part of the analysis of Masoudian et al. is based on a direct control of the distribution drift. The control is achieved by damping the learning rate to make sure that the played distribution on arms is not changing too much from round t , when an action is played, to round t + d t , when the loss is observed. Highly varying delays cannot be treated with this approach, because fast learning rates limit the range d t for which the drift is under control, while slow learning rates prevent learning. Therefore, Masoudian et al. had to reintroduce the assumption that that the maximal delay d max is known, and used it to tune the learning rate. Unfortunately, damping of the learning rate to control the drift over d max rounds made d max show up additively in the bound, meaning that potential presence of even a single delay of order T made both the stochastic and the adversarial bounds linear in the time horizon. We emphasise that the linear dependence of the regret on d max is real and not an artefact of the analysis, because it comes from damped learning rate.

We introduce a different best-of-both-worlds modification of the algorithm of Zimmert and Seldin (2020) that is fully parameterized by the running count of outstanding observations and requires no advance knowledge of delays or the maximal delay d max . Our algorithm is based on a careful augmentation of the algorithm of Zimmert and Seldin with implicit exploration (described below), followed by application of a skipping technique (also described below) as a tool to limit the time span over which we need to control the distribution shift.

Implicit exploration was introduced by Neu (2015) to control the variance of importance-weighted loss estimates in adversarial bandits. But the exploration parameters add up linearly to the regret bound, making it highly challenging to design a scheme for best-of-both-worlds setting. The implicit exploration schedule of Neu leads to Ω( √ T ) regret bound and, therefore, unsuitable for that. Jin et al. (2022) introduced a different schedule for adversarial Markov decision processes with delayed feedback. However, it is unknown whether their schedule can work in a stochastic analysis. We introduce a novel schedule and show that it works in best-of-both-worlds setting.

Skipping was introduced by Thune et al. (2019) as a way to limit the dependence of an algorithm on a small number of excessively large delays. The idea is that it is 'cheaper' to skip a round with an excessively large delay and bound the regret in the corresponding round by 1, than to include it in the core analysis. Thune et al. have assumed prior knowledge of delays, but Zimmert and Seldin (2020) have perfected the technique by basing it on a running count of outstanding observations. In both works skipping was an optional add-on aimed to improve regret bounds in case of highly unbalanced delays. In our work skipping becomes an indispensable part of the algorithm, because, apart from making the algorithm robust to a few excessively large delays, it also limits the time span over which the control of distribution drift is needed.

In Table 1 we compare our results to state of the art. In a nutshell, we replace terms dependent on d max by terms dependent on σ max , and terms dependent on the square root of the total cumulative delay D = ∑ T t =1 d t , by terms dependent on the number of skipped rounds |S| and a square root of the cumulative delay D ¯ S = ∑ t ∈ ¯ S d t in the non-skipped rounds ¯ S (those with the smaller delay). This yields robustness to excessive delays, because neither σ max nor min S ( |S| + √ D ¯ S ) depend on the magnitude of delay outliers. By contrast, both the stochastic and the adversarial regret bounds of

Masoudian et al. (2022) become linear in T in presence of a single delay of order T .

There are also additional benefits. It has been shown that σ max ≤ d max , and in some cases σ max ≪ d max (Joulani et al., 2013, Masoudian et al., 2022). For example, if the first observation has delay T , and the remaining observations have zero delay, then d max = T , but σ max = 1 . We also have that min S ( |S| + √ D ¯ S ) ≤ √ D , because S = ∅ is part of the minimization on the left, and in some cases min S ( |S| + √ D ¯ S ) ≪ √ D . For example, if the delays in the first √ T rounds are of order T , and the delays in the remaining rounds are zero, then min S ( |S| + √ D ¯ S ) = O ( √ T ) , but √

D = Ω ( T 3 / 4 ) (Thune et al., 2019). Therefore, bounds that exploit skipping are preferable over bounds that do not, and for some problem instances the improvement is significant. In Appendix F we show that bounds with an additive term d max , including the results of Masoudian et al. (2022), cannot benefit from skipping, in contrast to ours.

The following list highlights our main contributions.

We provide the first best-of-both-worlds algorithm for bandits with delayed feedback that is robust to delay outliers. It improves both the stochastic and the adversarial regret bounds relative to the work of Masoudian et al. (2022), which lacks such robustness. For some problem instances the improvement is dramatic, e.g., in presence of a single delay of order T both the stochastic and the adversarial regret bounds of Masoudian et al. are of order T , whereas our bounds are unaffected.

We provide an efficient technique to control the distribution drift under highly varying delays.

We provide the first implicit exploration scheme that works in best-of-both-worlds setting.

We provide a procedure relating drifted regret to normal regret in presence of delay outliers.

At the conceptual level, we show that best-of-both-worlds regret depends on the amount of information missing at the time of decision making (the number of outstanding observations) rather than the time that the information is missing (the delays). It was shown to be the case for the stochastic and adversarial regimes in isolation (Joulani et al., 2013, Zimmert and Seldin, 2020), but we are the first to show that it is also the case for best-of-both-worlds.

2 Problem setting

We study the problem of multi-armed bandit with variable delays. In each round t = 1 , 2 , . . . , the learner picks an action I t from a set of K arms and immediately incurs a loss ℓ t,I t from a loss vector ℓ t ∈ [0 , 1] K . However, the incurred loss is observed by the learner only after a delay of d t , at the end of round t + d t . The delays are arbitrary and chosen by the environment. We use σ t to denote the number of outstanding observations at time t defined as σ t = ∑ s ≤ t 1 ( s + d s > t ) and σ max = max t ∈ [ T ] σ t to be the maximal number of outstanding observations. We consider two regimes for generation of losses by the environment: oblivious adversarial and stochastic.

We use pseudo-regret to compare the expected total loss of the learner's strategy to that of the best fixed action in hindsight. Specifically, the pseudo-regret is defined as:

where i ∗ T = min i ∈ [ K ] E [ ∑ T t =1 ℓ t,i ] is the best action in hindsight. In the oblivious adversarial setting, the losses are assumed to be deterministic and independent of the actions taken by the algorithm. As a result, the expectation in the definition of i ∗ T can be omitted and the pseudo-regret definition coincides with the expected regret. Throughout the paper we assume that i ∗ T is unique. This is a common simplifying assumption in best-of-both-worlds analysis (Zimmert and Seldin, 2021). Tools for elimination of this assumption can be found in Ito (2021).

3 Algorithm

The algorithm is a best-of-both-worlds modification of the adversarial FTRL algorithm with hybrid regularizer by Zimmert and Seldin (2020). It is provided in Algorithm 1 display. The modification includes biased loss estimators (implicit exploration) and adjusted skipping threshold. The algorithm maintains a set of skipped rounds S t (initially empty), a cumulative count of 'active' outstanding observations (those that have not been skipped yet), and a vector of cumulative observed loss estimates ̂ L obs t from non-skipped rounds. At round t the algorithm constructs an FTRL distribution x t over arms using regularizer F t defined in equation (2) below, and samples an arm according to x t . Then it receives the observations that arrive at round t , except those that come from the skipped rounds, and updates the vector ̂ L obs t of cumulative loss estimates. The loss estimates ̂ ℓ t are defined below in equation (1). Then it counts the number of 'active' outstanding observations ̂ σ t (those that belong to non-skipped rounds), updates the cumulative count of outstanding observations D t , and computes the skipping threshold d t max = √ D t 49 K 2 / 3 log K . Finally, it adds rounds s for which the observation

has not arrived yet and the waiting time ( t -s ) exceeds the skipping threshold d t max to the set of skipped rounds S t . Lemma 20, which is an adaptation of Zimmert and Seldin (2020, Lemma 5) to our skipping rule, shows that at most one round s is skipped at a time (at most one index s satisfies the if-condition for skipping in Line 15 of the algorithm for a given t ).

We use implicit exploration to control importance-weighted loss estimates. The idea of using implicit exploration is inspired by the works of Neu (2015) and Jin et al. (2022), but its parametrization and purpose are different from prior work. To the best of our knowledge, it is the first time implicit exploration is used for best-of-both-worlds bounds. For any s, t ∈ [ T ] with s ≤ t we define implicit exploration terms λ s,t = e -D t D t -D s . Our biased importance-weighted loss estimators are defined by

where ̂ d s = min ( d s , min { ( t -s ) : t -s ≥ d t max } ) denotes the time that the algorithm waits for the observation from round s . It is the minimum of the delay d s , and the time ( t -s ) to the first round when the waiting time exceeds the skipping threshold d t max .

We use a hybrid regularizer based on a combination of the negative Tsallis entropy and the negative entropy, with separate learning rates,

where the learning rates are η -1 t = √ t and γ -1 t = √ 49 D t log K . The regularizer is the same as the one used by Zimmert and Seldin (2020). By inheriting their regularizer we inherit their adversarial regret bound, which is minimax optimal, with just a minor adjustment due to introduction of implicit exploration and a slight change in the learning rates and skipping threshold. The main contribution of our work is carrying out the stochastic analysis while staying within the algorithmic framework of Zimmert and Seldin and keeping the adversarial regret bound almost unscathed.

The update rule for x t is

where ̂ L obs t = ∑ t -1 s =1 ̂ ℓ s 1 ( s + d s < t ) 1 ( s / ∈ S t -1 ) is the cumulative importance-weighted loss estimate of observations that have arrived by time t and have not been skipped. We use S ∗ = S T to denote the final set of skipped rounds at time T .

4 Regret Bounds

The following theorem provides best-of-both-worlds regret bounds for Algorithm 1. A proof is provided in Section 5 and a bound on S ∗ can be found in Appendix H.

Theorem 1. The pseudo-regret of Algorithm 1 for any sequence of delays and losses satisfies

where ̂ σ max = max t ∈ [ T ] { ̂ σ t } is the maximal number of outstanding observations after skipping and

Furthermore, if the losses are stochastic, the pseudo-regret also satisfies

̸

Masoudian et al. (2022) provide an Ω ( √ KT +min S⊂ [ T ] { |S| + √ D ¯ S log K } ) regret lower bound for adversarial environments with variable delays, which is matched within constants by the algorithm of (Zimmert and Seldin, 2020) for adversarial environments. Our algorithm matches the lower bound within a multiplicative factor of K 1 3 on the delay-dependent term, which is the price we pay for obtaining a best-of-both-worlds guarantee. The price comes from a reduction of the skipping threshold of Zimmert and Seldin (2020) that we had to make to control the distribution drift that is due to the loss shift (see Appendix B.2). It is an open question whether this factor can be reduced.

̸

In the stochastic regime, assuming that the delays in the first σ max rounds are of order T , and that the losses come from Bernoulli distributions with bias close to 1 2 , a trivial regret lower bound is Ω ( σ max ∑ i = i ∗ ∆ i K + ∑ i = i ∗ log T ∆ i ) . This bound is almost matched by the algorithm of Joulani et al. (2013) for the stochastic regime only. Our bound has some extra terms, most notably ∑ i = i ∗ ̂ σ max ∆ i log K and S ∗ . It is an open question whether these terms are inevitable or can be reduced.

̸

Theorem 1 provides three major improvements relative to the results of Masoudian et al. (2022): (1) it requires no advance knowledge of d max ; (2) it replaces terms dependent on d max by terms dependent on ̂ σ max , which never exceeds d max , and in some cases may be significantly smaller; and (3) it makes skipping possible and beneficial, making the algorithm robust to a small number of excessively large delays and replacing √ D log K term with min S⊆ [ T ] { |S| + √ D ¯ S K 2 3 log K } , which is never much larger, but in some cases significantly smaller.

5 Analysis

In this section, we present a proof of Theorem 1. We begin with the stochastic part of the bound in Section 5.1, followed by the adversarial part in Section 5.2.

5.1 Stochastic Analysis

We start by defining the drifted regret Reg drift T = E [ ∑ T t =1 ( ⟨ x t , ̂ ℓ obs t ⟩ -̂ ℓ obs t,i ∗ T )] , where ̂ ℓ obs t = ∑ t s =1 ̂ ℓ s 1 ( s + ̂ d s = t ) 1 ( s / ∈ S t ) is the cumulative vector of losses received at time t . Lemma 2 is the first major contribution establishing a relationship between Reg drift T and the actual regret Reg T .

̸

Lemma 2 (Drift of the Drifted Regret) . Let σ t max = max s ∈ [ t ] { ̂ σ s } . Then

where S ∗ is the total number of rounds skipped by the algorithm.

The core of Lemma 2 is based on controlling the distribution drift using implicit exploration and skipping. In prior work on bounded delays the relation between Reg drift T and Reg T was achieved by shifting all the arrivals by d max , leading to an additive term of order d max . This approach fails for unbounded delays, because a single delay of order T prevents shifting and leads to linear regret. We address the challenge by introducing a procedure to rearrange the arrivals (Algorithm 2 below) and advanced control of the drift (Lemma 3 below). A proof of Lemma 2 is provided at the end of the section.

Algorithm 2: Greedy Rearrangement

The drift control lemma (Lemma 3) is the second major contribution of the paper. Prior work on bounded delays controlled the drift by slowing the learning rate in accordance with d max . This does not work for highly varying delays, because slow learning rates prevent learning, whereas fast learning rates fail to control the drift. Lemma 3 relies on implicit exploration terms in the loss estimators in equation (1) and on skipping of excessive delays, leaving the learning rates intact.

Lemma 3 (Drift Control Lemma) . Let d t max be the skipping threshold at time t . Then, for any i ∈ [ K ] and s, t ∈ [ T ] , where s ≤ t and t -s ≤ d t max , we have

The proof is based on introduction of an intermediate variable ˜ x s = ∇ ¯ F ∗ s ( -̂ L obs t -1 ) , which is based on the regularizer from round s and the loss estimate from round t . It exploits the implicit exploration term λ s,t to show that x t,i max( ˜ x i ,λ s,t ) ≤ 2 and skipping to show that ˜ x i x s,i ≤ 2 . The latter implies that max( ˜ x i ,λ s,t ) max( x s,i ,λ s,t ) ≤ 2 , and in combination with the former completes the proof. The details of the two steps are provided in Appendix B.

Given Lemmas 2 and Lemma 3, we apply standard FTRL analysis, similar to Masoudian et al. (2022), to obtain an upper bound for Reg drift T . Specifically, in Appendix A we show that

̸

̸

where a, b, c ≥ 0 are constants and υ t = ∑ t s =1 1 ( s + ̂ d s = t ) is the number of arrivals at time t (if a round s is skipped at time t it counts as an 'empty' arrival with loss estimate set to zero). By combining (4) with Lemma 2, we obtain

̸

̸

Then we apply a self-bounding analysis, similar to Masoudian et al. (2022), and get

̸

The details of the self-bounding analysis are provided in Appendix C.

The stochastic analysis is completed by the following lemma, which bounds the sum of implicit exploration terms above. It constitutes the third key result of the paper and shows that the bias from implicit exploration does not deteriorate neither the stochastic nor the adversarial bound. The proof is based on a careful study of the evolution of D t throughout the game, and is deferred to Appendix D.

Lemma 4 (Summation Bound) . For all s ∈ [ T ] , let D s = ∑ s r =1 ̂ σ r and λ s,t = e -D t D t -D s , then

Proof of Lemma 2 (Drift of the Drifted Regret)

We start with the definition of the drifted regret.

Note that when taking the expectation, we rely on the fact that ̂ ℓ s with s + ̂ d s = t does not affect x t . If max { x s,i , λ s,t } = x s,i , then ⋆ = ℓ s,i x t,i , otherwise

where the first inequality uses x t,i ≤ 4 max( x s,i , λ s,t ) = 4 λ s,t by Lemma 3, and ℓ s,i ≥ 1 , and the second inequality follows by x s,i ≥ 0 . Plugging (7) into (6) gives

It suffices to give a lower bound for R T in terms of the actual regret Reg T . The difference between R T and Reg T is that Reg T = E [ ∑ T t =1 ∑ K i =1 ∆ i x t,i ] , whereas in R T the sum ∑ K i =1 ∆ i x t,i is multiplied by the number of arrivals υ t = ∑ t s =1 1 ( s + ̂ d s = t ) at time t , and υ t might be larger than one or zero due to delays.

Our main idea here is to leverage the drift control lemma to provide a lower bound for R T in terms of Reg T . Specifically, by Lemma 3 for all r ∈ [0 , d t max ] , we have max( x t,i , λ t,t + r ) ≥ 1 4 x t + r,i , which implies x t,i ≥ 1 4 x t + r,i -λ t,t + r . Thus, we obtain the following bound for any r ∈ [0 , d t max ]

In Algorithm 2 we provide a greedy procedure to rearrange the arrivals by postponing some arrivals to future rounds to create a hypothetical rearranged sequence with at most one arrival at each round. Colliding arrivals are postponed to the first available (unoccupied) slot in the future. In Lemma 5 below we show that arrival originally received at time t stays in the [ t, t + σ t max ] interval (note that σ t max ≤ d t max ). When an observation from round s is postponed from arriving at round t to arriving at round t + r for r ∈ [0 , d t max ] , by (9) it is equivalent to replacing ∑ K i =1 ∆ i x t,i by 1 4 ∑ K i =1 ∆ i x t + r,i -Kλ t,t + r in R T . Note that Algorithm 2 may push an arrival to a round larger than T , which is equivalent to replacing ∑ K i =1 ∆ i x t,i by zero.

Let υ new t for all t ∈ [ T + d T max ] be the total arrivals at time t after the rearrangement, and let π ( t ) be the round to which we have mapped round t for all t ∈ [ T ] . Then for any rearrangement

The following lemma provides properties of the rearrangement procedure.

Lemma 5. Let σ t max = max s ∈ [ t ] { ̂ σ s } . Then Algorithm 2 ensures for any t ∈ [ T + d T max ] that υ new t ∈ { 0 , 1 } . Furthermore, for any round t ∈ [ T ] it keeps all the arrivals at time t in the interval [ t, t + σ t max ] , such that ∀ s ≤ t : s + ̂ d s = t ⇒ π ( s ) -t ≤ σ t max .

We provide a proof of the lemma in Appendix E. As a corollary, after the Greedy Rearrangement (Algorithm 2) the number of rounds with zero arrivals is at most σ T max . This is because there will be no arrivals after T + σ T max and ∑ T + σ T max t =1 υ new t = ∑ T t =1 υ t = T , which implies there are at most σ T zero arrivals as each round receives at most one arrival. Therefore max

where the first equality uses the definition of Reg T = E [ ∑ T t =1 ∑ K i =1 ∆ i x t,i ] and that ∀ t ∈ [ T ] : υ new t ∈ { 0 , 1 } .

Since ∀ t ∈ [ T ] : π ( t ) ≤ t + ̂ d t + σ t max , we have λ t,π ( t ) ≤ λ t,t + ̂ d t + σ t max . Together with (11), (10), and (8) it completes the proof.

5.2 Adversarial Analysis

The adversarial analysis is similar to the analysis of Zimmert and Seldin (2020, Theorem 2). In Appendix G we show that

where the first two terms originate from the analysis of Zimmert and Seldin due to structural similarity of the algorithm, S ∗ is due to adjusted skipping threshold, and K ∑ T t =1 λ t,t + ̂ d t is due to implicit exploration bias and is bounded by Lemma 4. The proof is completed by the following bound on S ∗ , which is shown in Appendix H.

6 Discussion

We have successfully addressed the challenge of handling varying and potentially unbounded delays in best-of-both-worlds setting. The success was based on three technical innovations, which may be interesting in their own right: (1) A relation between the drifted and the standard regret under unbounded delays (given by Lemma 2, Algorithm 2, and Lemma 5); (2) A novel control of distribution drift based on implicit exploration and skipping that does not alter the learning rates and exhibits efficiency under highly varying delays (Lemma 3); and (3) An implicit exploration scheme applicable in best-of-both-worlds setting (Lemma 4).

̸

The work leads to several directions for future research. One question is whether the best-of-bothworlds bounds could be improved further. In particular, whether the K 1 3 term in the adversarial regret bound could be reduced or eliminated. The term arose due to the need to decrease the skipping threshold of Zimmert and Seldin (2020) to control the distribution drift. It would also be valuable to explore whether it is possible to reduce the S ∗ term and reduce or eliminate the ∑ i = i ∗ ̂ σ max ∆ i log K term in the stochastic bound, or to derive lower bounds showing that these terms are unavoidable. Another interesting direction is to find more applications for implicit exploration and skipping in the context of best-of-both-worlds bounds.

Acknowledgments and Disclosure of Funding

SM and YS acknowledge partial support by the Independent Research Fund Denmark, grant number 9040-00361B.

References

Jacob D Abernethy, Chansoo Lee, and Ambuj Tewari. Fighting bandits with a new kind of smoothness. In Advances in Neural Information Processing Systems (NeurIPS) , 2015.

Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning , 47, 2002.

Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing , 32, 2002b.

Ilai Bistritz, Zhengyuan Zhou, Xi Chen, Nicholas Bambos, and Jose Blanchet. Online exp3 learning in adversarial bandits with delayed feedback. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Nicolo Cesa-Bianchi, Claudio Gentile, and Yishay Mansour. Delay and cooperation in nonstochastic bandits. Journal of Machine Learning Research , 20:1-38, 2019.

Stephen G. Eick. The two-armed bandit with delayed responses. The Annals of Statistics , 1988.

Shinji Ito. Parameter-free multi-armed bandit algorithms with hybrid data-dependent regret bounds. In Proceedings of the Conference on Learning Theory (COLT) , 2021.

Tiancheng Jin, Tal Lancewicki, Haipeng Luo, Yishay Mansour, and Aviv Rosenberg. Near-optimal regret for adversarial MDP with delayed bandit feedback. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Pooria Joulani, Andras Gyorgy, and Csaba Szepesvari. Online learning under delayed feedback. In Proceedings of the International Conference on Machine Learning (ICML) , 2013.

Saeed Masoudian, Julian Zimmert, and Yevgeny Seldin. A best-of-both-worlds algorithm for bandits with delayed feedback. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Gergely Neu. Explore no more: Improved high-probability regret bounds for non-stochastic bandits. In Advances in Neural Information Processing Systems , 2015.

Gergely Neu, András György, Csaba Szepesvári, and András Antos. Online markov decision processes under bandit feedback. In Advances in Neural Information Processing Systems , 2010.

Gergely Neu, András György, Csaba Szepesvári, and András Antos. Online markov decision processes under bandit feedback. IEEE Transactions on Automatic Control , 59:676-691, 2014.

Francesco Orabona. A modern introduction to online learning, 2022. https://arxiv.org/abs/ 1912.13213 .

Richard Simon. Adaptive treatment assignment methods and clinical trials. Biometrics , 33, 1977.

William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25, 1933.

Tobias Sommer Thune, Nicolò Cesa-Bianchi, and Yevgeny Seldin. Nonstochastic multiarmed bandits with unrestricted delays. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Julian Zimmert and Yevgeny Seldin. An optimal algorithm for adversarial bandits with arbitrary delays. In Proceedings on the International Conference on Artificial Intelligence and Statistics (AISTATS) , 2020.

Julian Zimmert and Yevgeny Seldin. Tsallis-INF: An optimal algorithm for stochastic and adversarial bandits. Journal of Machine Learning Research , 2021.

A Details of the Drifted Regret Analysis

In this section we prove the bound on drifted regret in equation (4). The derivation is same as the one by Masoudian et al. (2022), however, for the sake of completeness we reproduce it here. The analysis follows the standard FTRL approach, decomposing the drifted pseudo-regret into penalty and stability terms as

The penalty term is bounded by the following inequality, derived by Abernethy et al. (2015)

where e i ∗ T represents the unit vector in R K with the i ∗ T -th element being one and zero elsewhere. This leads to the following bound for penalty term

̸

where we substitute the explicit form of the regularizer into (12) and exploit the properties η -1 t -η -1 t -1 = O ( η t ) , γ -1 t -γ -1 t -1 = O ( σ t γ t / log K ) , and x 1 2 t,i ∗ T -1 ≤ 0 .

For the stability term, following a similar analysis as presented by Masoudian et al. (2022, Lemma 5), but incorporating implicit exploration terms, for any α t ≤ γ -1 t we obtain

Let A t = { s ≤ t : s + ̂ d s = t } , then due to the choice of skipping threshold, α t = ∑ s ∈ A t ¯ ℓ s,t satisfies the condition α t ≤ γ -1 t , where ¯ ℓ s,t = ∑ K i =1 f ′′ t ( x t,i ) -1 ̂ ℓ s,i ∑ i K =1 f ′′ t ( x t,i ) -1 = f ′′ t ( x t,I s ) -1 ̂ ℓ s,I s ∑ i K =1 f ′′ t ( x t,i ) -1 . Thus we have

̸

For brevity we define z t,i = f ′′ t ( x t,i ) -1 and m t s,i = max { x s,i , λ s,t } for any s ≤ t and i ∈ [ K ] . We begin bounding S 1 by replacing definition of loss estimators from (1) and get

Where the first inequality uses ℓ s,I s ≤ 1 . We show that S 2 1 has negative contribution to S 1 by taking expectation w.r.t. I s as the following

Thus we only need to bound S 1 1 , for which we take expectation w.r.t. I s and separate i ∗ from the other arms to get

̸

̸

where the second inequality uses z t,i = f ′′ t ( x t,i ) -1 ≤ η t x 3 / 2 t,i along x t,i ≤ m t s,i from Lemma 3, the third inequality is due the fact that z t,i ∗ ( 1 -z t,i ∗ ∑ K j =1 z t,j ) is an increasing function in terms

̸

̸

̸

̸

̸

̸

̸

̸

of both z t,i ∗ and ∑ i = i ∗ z t,i and we substitute z t,i ∗ ≤ η t x 3 / 2 t,i ∗ and ∑ j = i ∗ z t,j ≤ ∑ j = i ∗ η t x 3 / 2 t,j ≤ η t (1 -x t,i ∗ ) 3 / 2 , the fourth inequality is due to (1 -a ) 3 / 2 + a 3 / 2 ≤ 2 -1 / 2 , the fifth and the sixth inequalities rely on Lemma 3, and finally the last inequality is followed by ∀ i : x s,i ≤ x 1 / 2 s,i and that η t ≤ η s . Combining bounds for S 1 1 and S 2 1 gives the following bound for S 1

̸

̸

̸

For S 2 , we take expectation with respect to I s , I r , and randomness of losses, all separately to get

̸

̸

For simplicity we define ϵ t s,i = µ i -µ i x s,i m t s,i for any s ≤ t and any i ∈ [ K ] , for which we have the following bounds

We then continue from 15 and bound it as the following E [ S 2 ]

̸

where the inequality holds because we ignore the negative terms after multiplication and that | ( µ i -∑ K j =1 z t,j µ j ∑ K j =1 z t,j ) | ≤ 1 . We need to bound each part from (16). We start with S 1 2 ,

̸

̸

We bound S 2 2 as

̸

̸

̸

where the last inequality holds because z t,i ≤ γ t x t,i and that x t,i ≤ 4 m t s,i , 4 m t r,i from Lemma 3. It remains to give upper bound for S 3 2 as

where the second inequality rely on z t,i ≤ γ t x t,i , λ s,t ≤ m t s,i , λ r,t ≤ m t r,i , and x t,i ≤ 4 m t s,i , x t,i ≤ 4 m t r,i from Lemma 3. It is suffices to plug bounds in (17), (18), and (19) to obtain

̸

where the third inequality uses Lemma 3 and the last inequality holds because of the skipping that ensures γ t + ̂ d t ( υ t + ̂ d t -1) ≤ 1 . Now, it is sufficient to combine the bounds for S 1 and S 2 in (14) and (20) and get

̸

̸

Combining the stability bound from (21) and the penalty bound from (13) concludes the proof.

B Proof of the Drift Control Lemma

In this section we provide a proof of Lemma 3. We start with a few auxiliary results, and then prove the lemma.

B.1 Auxiliary results for the proof of the key lemma

For the proof we use two facts and a lemma from Masoudian et al. (2022), and a new lemma. Recall that f t ( x ) = -2 η -1 t √ x + γ -1 t x log x .

Fact 7. (Masoudian et al., 2022, Fact 15) f ′ t ( x ) is a concave function.

Fact 8. (Masoudian et al., 2022, Fact 16) f ′′ t ( x ) -1 is a convex function.

Lemma 9. (Masoudian et al., 2022, Lemma 17) Fix t and s with t ≥ s , and assume that there exists α , such that x t,i ≤ α max( x s,i , λ s,t ) for all i ∈ [ K ] , and let f ( x ) = ( -2 η -1 t √ x + γ -1 t x log x ) , then we have the following inequality

Lemma 10. If t > s and ( t -s ) ≤ d t max , then

which is equivalent to D t ≤ 2 D s .

Proof. It suffices to prove that D t ≤ 2 D s , which is equivalent to proving that ( D t -D s ) ≤ 1 2 D t . We have:

where the first inequality holds because due to skipping, for all r ≤ t we have ̂ σ r ≤ d t max , and ( t -s ) ≤ d t max .

B.2 Proof of the Drift Control Lemma

Now we are ready to provide a proof of Lemma 3. Similar to the analysis of Masoudian et al. (2022), the proof relies on induction on valid pairs ( t, s ) , where a pair ( t, s ) is considered valid if s ≤ t and ( t -s ) ≤ d t max . The induction step for pair ( t, s ) involves proving that x t,i ≤ 4 max( x s,i , λ s,t ) for all i ∈ [ K ] . To establish this, we use the induction assumption for all valid pairs ( t ′ , s ′ ) such that s ′ , t ′ < t , as well as all valid pairs ( t ′ , s ′ ) , such that t ′ = t and s < s ′ ≤ t . The induction base encompasses all pairs ( t ′ , t ′ ) for all t ′ ∈ [ T ] , where the statement x t ′ ,i ≤ 4 x t ′ ,i holds trivially.

To control x t,i max( x s,i ,λ s,t ) we first introduce an auxiliary variable ˜ x = ¯ F ∗ s ( -̂ L obs t -1 ) . We then address the problem of drift control by breaking it down into two sub-problems:

x t,i max( ˜ x i ,λ s,t ) ≤ 2 : the drift due to change of regularizer,

˜ x i x s,i ≤ 2 : the drift due to loss shift.

Deviation induced by the change of regularizer

The regularizer at round r is defined as

We have x t = ∇ ¯ F ∗ t ( -̂ L obs t -1 ) and ˜ x = ∇ ¯ F ∗ s ( -̂ L obs t -1 ) . According to the KKT conditions, there exist Lagrange multipliers µ and ˜ µ , such that for all i :

We also know that there exists an index j , such that ˜ x j ≥ x t,j . This leads to the following inequality:

where the first inequality holds because the learning rates are decreasing, and the second inequality is due to the fact that f ′ s ( x ) is increasing. This implies that µ ≤ ˜ µ , which gives us the following inequality for all i :

Thus, we have two cases, either -1 η t √ x t,i ≤ -1 η s √ ˜ x i or log( x t,i ) γ t ≤ log( ˜ x i ) γ s .

Case i: If -1 η t √ x t,i ≤ -1 η s √ ˜ x i holds, then we have x t,i ˜ x i ≤ η 2 s η 2 t = t s . On the other hand, we have

where the second inequality holds because trivially ̂ σ r ≤ r . This implies that x t,i ˜ x i ≤ 2 .

Case ii: If log( x t,i ) γ t ≤ log( ˜ x i ) γ s , it implies that x t,i ≤ x γ t γs i . Using x i ≤ max( x i , λ s,t )

Therefore, in both cases we obtain

Deviation Induced by the Loss Shift

The initial steps of the proof of this part are the same as in Masoudian et al. (2022). However, for the sake of completeness, we restate them here.

Since we have x s = ∇ ¯ F ∗ s ( -̂ L obs s -1 ) and ˜ x = ∇ ¯ F ∗ s ( -̂ L obs t -1 ) , they both share the same regularizer F s ( x ) = ∑ K i =1 f s ( x i ) . For brevity, we drop s from f s ( x ) . By the KKT conditions ∃ µ, ˜ µ s.t. ∀ i :

Let ˜ ℓ = ̂ L obs t -1 -̂ L obs s -1 , then by the concavity of f ′ ( x ) from Fact 7, we have

Since f ′′ ( x s,i ) ≥ 0 , from the left side of (23) we get x s,i -˜ x i ≤ f ′′ ( x s,i ) -1 ( µ -˜ µ + ˜ ℓ i ) . Taking summation over all i and using the fact that both vectors x s and ˜ x are probability vectors, we have

Combining the right hand sides of (23) and (24) gives

and by rearrangement we get

where the last inequality holds because f ′′ ( ˜ x i ) -1 = ( η -1 s 1 2 ˜ x -3 / 2 i + γ -1 s ˜ x -1 i ) -1 . The next step for bounding ˜ x i is to bound ∑ K j =1 f ′′ ( x s,j ) -1 ˜ ℓ j ∑ K j =1 f ′′ ( x s,j ) -1 in (25), where ˜ ℓ j = ∑ r ∈ A ̂ ℓ r,j and A = { r : s ≤ r + ̂ d r < t } .

If there exists r ∈ A , such that r > s and 4 max( x r,i , λ r,r + ̂ d r ) ≤ x s,i , then combining it with the induction assumption for ( r + ̂ d r , r ) , where we have x r + ̂ d r ,i ≤ 4 max( x r,i , λ r,r + ̂ d r ) , leads to x r + ̂ d r ,i ≤ x s,i . On the other hand, by the induction assumption for pair ( r + ̂ d r , t ) , we have

So using x r + ̂ d r ,i ≤ x s,i and λ r + ̂ d r ,t ≤ λ s,t we can derive x t,i ≤ 4 max( x s,i , λ s,t ) . This inequality satisfies the condition we wanted to prove in the drift lemma. Therefore, we assume that for all r ∈ A we have either r ≤ s or x s,i ≤ 4 max( x r,i , λ r,r + ̂ d r ) . If r ≤ s , using the the induction assumption for ( s, r ) together with the fact that λ r,s ≤ λ r,r + ̂ d r , results in x s,i ≤ 4 max( x r,i , λ r,s ) . Consequently, in either case, the following inequality holds for all r ∈ A

Thus, inequality in (26) satisfies the condition of Lemma 9, and for all r ∈ A we get:

We proceed by summing both sides of the inequality (27) over all r ∈ A and obtain ∑ K j =1 f ′′ ( x s,j ) -1 ˜ ℓ j ∑ K j =1 f ′′ ( x s,j ) -1 ≤ 4 | A | ( K -1) 1 3 . Now it suffices to plug this result into (25):

where the third inequality uses | A | ≤ d s max + t -s ≤ d t max + d s max , and that d t max ≤ 2 d s max by Lemma 10, and for the last inequality we use the definitions of γ s and d s max .

Combining (29) and (22) completes the induction step.

C Self-Bounding Analysis

In this section we show the details of how to apply self-bounding analysis to bound the right hand side of (5).

We start from (5) and decompose it as follows

̸

̸

D

We rewrite the pseudo-regret as Reg T = 4 Reg T -3 Reg T , and then based on the decomposition above we have

Masoudian et al. (2022) provide the following three lemmas that give the bounds for the first three terms in (30). Although the algorithm of Masoudian et al. differs from ours, their bounds remain applicable, because they are based on the worst-case choice of x t,i , which is algorithm-independent.

Lemma 11. (Masoudian et al., 2022, Lemma 6) For any a ≥ 0 , we have:

̸

Lemma 12. (Masoudian et al., 2022, Lemma 7) Let υ max = max t ∈ [ T ] υ t , then for any b ≥ 0 :

It is evident that υ max ≤ σ max , so the bound in Lemma 12 is dominated by O ( Kσ max ) term in the regret bound.

Lemma 13. (Masoudian et al., 2022, Lemma 8) For any c ≥ 0 :

̸

By plugging (31),(32),(33) into (30) we get the desired bound.

D A Proof of Lemma 4

First we provide two facts and two auxiliary lemmas.

Lemma 14. For any t we have

Proof. We show that for any t ∈ [ T ] we have ∑ t s =1 ̂ d s -D t ≤ D t :

where the second inequality holds because ̂ d s ≤ d t max , and the total number of steps that satisfy ( s ≤ t ) ∧ ( s + ̂ d s > t ) is less than the skipping threshold at time t , which is again d t max . Rearranging the inequality completes the proof.

Lemma 15 ((Orabona, 2022, Lemma 4.13)) . Let a 0 ≥ 0 and f : [0; + ∞ ) → [0; + ∞ ) be a nonincreasing function. Then

Fact 16. For any x ≥ 0 , we have e -x ≤ 1 x .

Fact 17. For any x ≥ 1 , we have e -x ≤ 1 x log 2 ( x ) .

Proof of Lemma 4. We have two summations as

where we show an upper bound of O ( ̂ σ max ) for each of them.

Bounding the First Summation: Let T 0 be the time satisfying √ D T 0 = ̂ σ max K 1 / 3 log( K ) , then using Facts 16 and 17 we have

For A we give the following bound

where the second equality is by swapping the summations, the first inequality holds because D t + ̂ d t ≥ D s , the third inequality uses ̂ σ s ≤ d s max ≤ √ D s K 1 / 3 log K , and the last inequality uses Lemma 15.

The bound for B is as follows

where the first inequality follows by ̂ σ s ≤ ̂ σ max and our skipping procedure that ensures ̂ d t ≤ d t max ≤ √ D t + ̂ d t K 1 / 3 log K , the second equality is by swapping the summations, the second inequality follows by D t + ̂ d t ≥ D s and ∑ s -1 t =1 1 ( t + ̂ d t ≥ s ) = ̂ σ s , the last inequality follows by Lemma 15 , and the last equality uses ∫ 1 x log 2 ( x/ ̂ σ 2 max ) dx = -1 log( x/ ̂ σ 2 max ) .

Bound the Second Summation: The bound for the second summation follows the same approach, but it requires additional care due to existence of σ t max in it. Let T 0 to be the time satisfying √ D T 0 = ̂ σ max K 1 / 3 log( K ) , then using Facts 16 and 17 we have

For A we give the following bound

where the first inequality is by Fact 16, the second inequality holds by swapping the summations and that D t + σ t max + ̂ d t ≥ D s , third inequality use the following derivation

the third equality is by swapping the summations, the third inequality uses ̂ σ s ≤ d s max ≤ √ D s K 1 / 3 log K , and finally the last inequality uses Lemma 15.

The bound for B is as follows

where the first inequality is due to our skipping procedure that ensures max { σ t max , ̂ d t } ≤ d t max ≤ √ D t + σ t max + ̂ d t , the second equality is by swapping the summations, the second inequality follows by D t + ̂ d t ≥ D s and (34), the last inequality follows by Lemma 15, and the last equality uses ∫ 1 x log 2 ( x/ ̂ σ 2 max ) dx = -1 log( x/ ̂ σ 2 max ) .

E A proof of Lemma 5

Proof. We use the term free round to refer to a round r such that υ new r is zero. By applying induction on the time step t , we show that if the algorithm is currently at time t and intends to rearrange the υ t arrivals, there exist υ t free rounds in the interval [ t, t + σ t max -̂ σ t + υ t ] to which the algorithm can push the arrivals. This ensures that the arrival from round s , will be rearranged to round π ( s ) ≥ s + ̂ d s , such that π ( s ) -( s + ̂ d s ) ≤ σ t max . To this end, we assume the induction assumption holds for all r < t , and then proceed with induction step for t .

Induction Base:

The induction base corresponds to the first arrival time, denoted as t 0 . At this time step, all υ t 0 arrivals can be rearranged to the free rounds in the interval [ t 0 , t 0 + υ t 0 -1] , which is a subset of [ t 0 , t 0 + σ t 0 max -̂ σ t 0 + υ t 0 -1] . Therefore, the induction base holds.

Induction step:

Assume that we are at round t , and our aim is to rearrange the arrivals of round t . We define t 1 as the last occupied round, where t 1 ≥ t . So it suffices to prove t 1 -t ≤ σ t max -̂ σ t . We first note that since the algorithm is greedy, all rounds t, t +1 , . . . , t 1 -1 must also be occupied by some arrivals from the past.

Let t 0 < t be the first round where one of its arrivals has been rearranged to t , and let υ ′ t 0 be the number of arrivals at time t 0 that are rearranged to some rounds before t . Then by induction assumption we know

On the other hand, by the choice of t 0 , each occupied round t, t +1 , . . . , t 1 must be occupied by exactly one arrival among the arrivals of rounds t 0 , . . . , t -1 , except for the υ ′ t arrivals of t 0 that are rearranged to some rounds before t . So we have

where the second equality holds because ∑ t -1 r = t 0 1 ( r + ̂ d r ≥ t 0 ) = t -t 0 . We use (35) to bound t -t 0 in the above inequality and get

where the last inequality follows by the fact that { σ r max } r ∈ [ T ] is a non-decreasing sequence. So if the algorithm rearranges the υ t arrivals at round t to rounds t 1 +1 , . . . , t 1 + υ t , then, using the inequality (36), we can conclude that these rounds fall within the interval [ t, t + σ t max -̂ σ t + υ t ] .

F Adversarial bounds with d max cannot benefit from skipping

In this section we show that adversarial regret bounds that involve terms that are linear in d max , such as the bounds of Masoudian et al. (2022), cannot benefit from skipping. We prove the following lemma.

Lemma 18.

Proof. For any split of the rounds [ T ] into S and ¯ S we have

Thus

and since the above holds for any S , we obtain the statement of the lemma.

We remind that skipping allows to replace a term of order √ D by a term of order min S ( |S| + √ D ¯ S ) (for simplicity we ignore factors dependent on K ). Thus, it may potentially replace a bound of order √ D + d max by a bound of order min S ( |S| + √ D ¯ S ) + d max , but since by the lemma min S ( |S| + √ D ¯ S ) + d max = Ω( √ D ) , this would not improve the order of the bound.

G Details of the Adversarial Analysis

The only difference between our algorithm and the algorithm of Zimmert and Seldin (2020) is the implicit exploration and the slightly modified skipping rule. Let ℓ t be the original loss sequence, then the adversary can create an adaptive sequence ˜ ℓ t that forces the player to play according to the implicit exploration rule by simply down-scaling all the losses by

Our regret bound decomposes now into

For the second term we have

which can be controlled via Lemma 4.

The first term is bounded by Zimmert and Seldin (2020, Theorem 3) (since the player plays their algorithm on the modified loss sequence) by

where the first equality uses the definition of γ t , the third inequality follows by ∀ a, b > 0 : a -b √ a ≤ 2( √ a -√ b ) , and the last inequality uses the following lemma

Lemma 19. The skipping technique guarantees the following bound

Combining the bounds on the first and the second terms provides the regret bound in Section 5.2. It only remains to provide a proof for Lemma 19.

Proof of Lemma 19. For any t ∈ [ T ] we have ̂ d t ≤ √ D T / (49 K 2 3 log( K )) , therefore for any R ⊂ [ T ] :

Hence we can dereive the following lower bound,

where the second inequality uses the concavity in r .

H A Bound on S ∗

Next, we reason about the nature of skips. The following lemma is an adaptation of Zimmert and Seldin (2020, Lemma 5) to our skipping threshold. To this end we provide two lemmas and then conclude then proof.

Lemma 20. Algorithm 1 will not skip more than 1 point at a time.

Proof. We prove the lemma by contradiction. Assume that s 1 , s 2 are both deactivated at time t . W.l.o.g. let s 2 ≤ s 1 -1 . Skipping of s 1 at time t means t -s 1 ≥ √ D t / ( K 2 3 log( K )) ≥ √ D t -1 / ( K 2 3 log( K )) . At the same time we assumed t -1 -s 2 ≥ t -s 1 , which means that s 2 would have been deactivated at round t -1 or earlier.

Recall that ̂ d t is the contribution of a timestep t to the sum D T . Let ( t 1 , . . . , t S ∗ ) be an indexing of S and c = 49 K 2 3 log( K ) . We bound the number of skips by

The above bound together with the fact that incurred delay ̂ d t ∗ S must be less than the the skipping threshold and the maximal delay d max give us

where the last inequality follows by Lemma 19.

Proof of bound (37) . By Lemma 20 we skip at most one outstanding observation per round. Thus, we have that

By solving the quadratic inequality in ̂ d t m we obtain

Now we prove by induction that ̂ d t m ≥ m 2 c . The induction base holds since ̂ d t 1 = 1 . For the inductive step we have

Then the induction step is satisfied.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We list all our contributions in the abstract and introduction, and included a 548 literature review in the introduction to reflect the scope

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We present all our bounds in Table 1 and discuss about the gap between our results and existing lower and upper bounds in the introduction section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Theorems state their assumptions explicitly. Proofs are provided concisely within the main body, with full details in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This work is based on theoretical analysis.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This work does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This work does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This work does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This work does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm that we adhere to ethical standards where applicable, and we note that no experiments were conducted as part of this research.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper focuses on theoretical foundations, and discussing potential societal impacts, positive or negative, is beyond its scope.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the best-of-both-worlds (BOBW) algorithms for MAB with delayed feedback. Compared to previous results, it eliminates the need for prior knowledge on maximum delay $d_{\\text{max}}$, and the regret scales with the number of “outstanding observations” ($\\sigma_{\\text{max}}$) rather than with the delay length as in the previous works. Technique-wise, this is achieved through a new round skipping rule, a new implicit exploration loss estimator, and advanced analysis.\n\nSTRENGTHS:\nThe regret guarantee is significant in the sense that it scales with the number of missing information, rather than how much it is delayed as in previous work. In the latter case, even if there’s only one single delay, it incurs linear regret if this delay is $T$, which however is safeguarded by the regret guarantee in this paper. Such phenomenon has been achieved soly in either stochastic or adversarial regime before, but not in BOBW.  To achieve this, the authors propose new techniques in both algorithm design and regret analysis.\n\nWEAKNESSES:\n1. I feel that the words \"distribution drift/shift\" comes from nowhere in the introduction. Its meaning is not clear to me in the context of delayed bandits, but no sufficient explanation is provided. A reader may not figure it out until the Analysis Section.\n2. The authors use both “multiarm” and “multi-arm” inconsistently. The latter is commonly used.\n3. The contribution summary may need polishing. Points 2-4 (Line 118) seems too  detailed. Currently they are not informative, they could be merged, or not even necessarily included in the contribution summary in my opinion.\n4. The author may consider elaborating more on Line 181-183, i.e., why the additional $K^{1/3}$ factor is a price for BOBW guarantee (in the analysis). This could be an interesting point to BOBW researchers.\n5. When introducing Lemma 2 for the first time (around Line 202), it would be good to briefly talk about what are the key elements to achieve it, rather than let the readers figure it out in the long proof later. Is it due to the analysis (virtual rearrangement) only, or also the skipping rule and/or implicit exploration?\n6. These two works on feedback-delayed bandits should be relevant, but they are not discussed in this paper. One in stochastic regime [1] and the other in adversarial regime [2].\n\nReferences:\n\n[1] Yang, Yunchang, Han Zhong, Tianhao Wu, Bin Liu, Liwei Wang, and Simon Shaolei Du. \"A Reduction-based Framework for Sequential Decision Making with Delayed Feedback.\" In Thirty-seventh Conference on Neural Information Processing Systems. 2023.\n\n[2] van der Hoeven, Dirk, Lukas Zierahn, Tal Lancewicki, Aviv Rosenberg, and Nicoló Cesa-Bianchi. \"A Unified Analysis of Nonstochastic Delayed Feedback for Combinatorial Semi-Bandits, Linear Bandits, and MDPs.\" In The Thirty Sixth Annual Conference on Learning Theory, pp. 1285-1321. PMLR, 2023.\n\nQUESTIONS:\n1. Why do some citations have no year? E.g., “analysis of Zimmert and Seldin” in Line 58\n\n2. Is the word “outstanding observations” existing in the delayed-feedback literature or named for the first time by the authors? It is just the number of missing observations, right? If so, I personally just feel that the wording is not very informative/natural.\n\n3. Is the idea of “skipping” reflected only through the regret analysis, or both the algorithm design and analysis? From Algorithm 1, I think it is the latter, but Line 86-89 seems to mean the former.\n\n4. Since the BOBW guarantee is obtained through the self-bounding trick, can we directly obtain a regret bound for the intermediate $C$-corrupted regime?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new best-of-both-world algorithm for bandits with a delayed feedback model. The results simultaneously achieve the latest upper bounds for delayed feedback in stochastic and adversarial (up to a factor). The algorithm design utilizes an implicit exploration estimator and the skipping set. The analysis contains new technical methods for handling the relationship between drifted and standard regrets.\n\nSTRENGTHS:\nThe reviewer thinks this paper is a nice piece of theoretical work. It has several strengths as follows, \n1. This paper's introduction concisely and clearly discusses this work and its relation (both result and theoretical tool aspects) to previous work. This is very helpful for readers to understand the research background and new contribution.\n2. The best-of-both-world theoretical results of this paper combine the known best stochastic and adversarial upper bounds (with a $K^{1/3}$) factor for bandits with delay and improve upon the latest results on the same topic.\n\nWEAKNESSES:\nThe reviewer is uncertain whether this paper reaches the NeurIPS bar for the following weaknesses reasons: \n1. The algorithm design itself needs to be more novel. The only new component the reviewer notices is the implicit exploration estimator, which was also related to prior works in adversarial bandits.  \n2. The algorithm design is not optimized. For example, the skipping set $\\mathcal{S_{t}}$ is always expanding, which is counterintuitive: (1) As the threshold $d^t_{\\text{max}}$ is non-decreasing, there should exist observations, previous belongs to the skipping set, with delay less than the increased new threshold. Why does the algorithm design not add these skipped observations to the non-skipped set? (2) Even for observations belonging to the skipping set, when their observations are finally revealed, why does the algorithm not consider the kind of observations? If considered by the decision-making, both types of observations can help improve the algorithm's performance. Even if this modification cannot reduce the regret upper bound, it should be able to improve the algorithm's empirical performance.\n\nQUESTIONS:\n1. The reviewer needs clarification on some proof steps. For example, by the definition of the drift regret, the benchmark is the estimated loss $\\hat \\ell_{i^*}$, but when it comes to the stability-penalty decomposition at the beginning of Appendix A, the benchmark becomes the actual loss $\\ell_{i^*}$, which looks inconsistent.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the best of both world algorithms for arbitrary delayed feedback. The proposed algorithm does not require prior knowledge of maximum delay $d_{\\max}$ and avoids its linear dependence in the regret bound. To this end, they proposed the implicit exploration that works for the best-of-both-worlds guarantees. The key idea is to relate the number of outstanding observations $\\sigma_{\\max}$ so as not to rely on the boundness of delays.\n\nSTRENGTHS:\n- The first proposal of implicit exploration for BoBW settings. The algorithm is based on FTRL with a hybrid regularizer where a skipping technique is employed, which becomes crucial for the BoBW guarantee. \n\n- When the maximal number of outstanding observations $\\sigma_{\\max}$ is smaller than maximum delay $d_{\\max}$, the proposed method provided a much better bound than [Masoudian et al. 2022].\n\nWEAKNESSES:\nI only see it for writing.  \nThe current main paper more focus on the explanation of the result comparison with [Masoudian et al. 2022]. Although the authors detailed a summary of technical proof in the main text, a more intuitive, and higher-level explanation of algorithmic parts would be appreciated when introducing the proposed technical scheme. For example, the paper does not provide a detailed explanation of the selection of regularizers or learning rates intuitively. For example, what makes analysis challenging due to aiming BoBW bounds could be more highlighted. I guess the additional challenge for dealing with stochastic cases, from [Zimmer and Seldin, 2020] that already uses the FTRL algorithm with a hybrid regularizer, has already been addressed by [Masoudian et al. 2022], but by missing these explanations, it would be hard for readers to see the clear picture of inherently technical challenges of BoBW algorithm for delayed feedback.\n\nQUESTIONS:\nHow can we interpret implicit exploration term $\\lambda_{s,t}$? (Is this inspired by controlling the bias term due to an implicit exploration scheme in order to make it applicable to a best-of-both-worlds setting?)\n\nLIMITATIONS:\nn.a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the multi-armed bandit problem with delayed feedback, where the loss of a chosen arm is observed several rounds later. In this setting, nearly optimal algorithms have been developed for both stochastic and adversarial environments, as well as best-of-both-worlds (BOBW) algorithms that perform well in both regimes. However, the regret upper bounds of existing BOBW algorithm by Masoudian et al. (2022) has gaps compared to the regret upper bounds of algorithms designed specifically for adversarial environments (Zimmert and Seldin, 2020) and stochastic environments (Joulani et al., 2013) and in particular the algorithm by Masoudian et al. require the prior knowledge of the maximal delay before the game starts.\nTo address this issue, the authors propose an algorithm that achieves a regret upper bound without knowing the maximal delay in advance.\nTo achieve this, the authors introduce a new framework of implicit exploration, demonstrating that this framework can be effectively combined with techniques that skip observations associated with excessive delays.\n\nSTRENGTHS:\nThe differences from existing research are thoroughly discussed. Technically, the introduction of the implicit exploration framework to achieve BOBW regret bounds without using the maximal delay is novel. Fine-tuned adjustments are made to achieve BOBW while employing implicit exploration. Additionally, how the parameter $\\lambda$ is utilized in the proof is clearly explained and discussed throughout the proof.\n\nWEAKNESSES:\nOne concern is that the differences between this paper and existing research might not be sufficient. While the authors propose an algorithm that works in both adversarial and stochastic environments, the foundation for handling excessive delays seems to have been largely established by Zimmert and Seldin (2020) (and Masoudian (2022)).\n\nI was unable to check all the proofs in detail, but they appear to be generally correct. However, there seems to be large room for improvement in the overall explanation. Many explanations are not contextualized, making the proofs difficult to read.\nFor instance, Lemmas 11 to 13 are directly taken from Masoudian et al. (2022). However, the algorithm in Masoudian et al. (2022) differs from the one proposed by the authors, so it is unclear whether these results are directly applicable.\n\nQUESTIONS:\nThe authors are expected to discuss to what extent the results from Zimmert and Seldin (2020) and Masoudian et al. (2022) can be used in their study.\n\nLIMITATIONS:\nnot applicable", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}