Title: Uniform Last-Iterate Guarantee for Bandits and Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: uniform last-iterate

bandits

reinforcement learning
Abstract: Existing metrics for reinforcement learning (RL) such as regret, PAC bounds, or uniform-PAC (Dann et al., 2017), typically evaluate the cumulative performance, while allowing the play of an arbitrarily bad policy at any finite time t. Such a behavior can be highly detrimental in high-stakes applications. This paper introduces a stronger metric, uniform last-iterate (ULI) guarantee, capturing both cumulative and instantaneous performance of RL algorithms. Specifically, ULI characterizes the instantaneous performance since it ensures that the per-round suboptimality of the played policy is bounded by a function, monotonically decreasing w.r.t. (large) round t, preventing revisits to bad policies when sufficient samples are available. We demonstrate that a near-optimal ULI guarantee directly implies near-optimal cumulative performance across aforementioned metrics, but not the other way around. 
To examine the achievability of ULI, we first provide two positive results for bandit problems with finite arms, showing that some elimination-based algorithms and high-probability adversarial algorithms with stronger analysis or additional designs, can attain near-optimal ULI guarantees. We also provide a negative result, indicating that optimistic algorithms cannot achieve a near-optimal ULI guarantee. Furthermore, we propose an efficient algorithm for linear bandits with infinitely many arms, which achieves the ULI guarantee, given access to an optimization oracle. Finally, we propose an algorithm that achieves a near-optimal ULI guarantee for the online reinforcement learning setting.

Uniform Last-Iterate Guarantee for Bandits and Reinforcement Learning

Abstract

Existing metrics for reinforcement learning (RL) such as regret, PAC bounds, or uniform-PAC [Dann et al., 2017], typically evaluate the cumulative performance, while allowing the agent to play an arbitrarily bad policy at any finite time t . Such a behavior can be highly detrimental in high-stakes applications. This paper introduces a stronger metric, uniform last-iterate (ULI) guarantee, capturing both cumulative and instantaneous performance of RL algorithms. Specifically, ULI characterizes the instantaneous performance by ensuring that the per-round suboptimality of the played policy is bounded by a function, monotonically decreasing w.r.t. round t , preventing revisiting bad policies when sufficient samples are available. We demonstrate that a near-optimal ULI guarantee directly implies near-optimal cumulative performance across aforementioned metrics, but not the other way around. To examine the achievability of ULI, we first provide two positive results for bandit problems with finite arms, showing that elimination-based algorithms and high-probability adversarial algorithms with stronger analysis or additional designs, can attain near-optimal ULI guarantees. We also provide a negative result, indicating that optimistic algorithms cannot achieve near-optimal ULI guarantee. Furthermore, we propose an efficient algorithm for linear bandits with infinitely many arms , which achieves the ULI guarantee, given access to an optimization oracle. Finally, we propose an algorithm that achieves near-optimal ULI guarantee for the online reinforcement learning setting.

1 Introduction

In online decision-making problems with bandit feedback, a learner sequentially interacts with an unknown environment: in each round, the learner plays an policy and then observes the corresponding rewards of the played policy. Typically, the goal of the learner is to achieve good cumulative performance , commonly measured by regret or probably approximately correct (PAC) bound. For instance, in the online advertisement scenario, the goal of the website (learner) could be maximizing the cumulative click numbers [Li et al., 2010]. Hence, the website aims to minimize the regret that measures the cumulative clicks of the recommended advertisement compared to that of the unknown optimal advertisement. In addition to regret minimization, the goal could also be quickly identifying popular advertisements [Chen et al., 2014, Jin et al., 2019]. To this end, a PAC bound

is suitable here to measure the sample complexity (i.e., cumulative time steps) that the algorithm needs to identify those popular advertisements. To reap the benefits of both measures, Dann et al. [2017] propose a new performance measure called uniform-PAC, ensuring that for all ϵ > 0 , the total number of ϵ -suboptimal policies played by the algorithm is bounded by a function polynomial in 1 /ϵ . The uniform-PAC bound can simultaneously imply a high-probability sublinear regret bound and a polynomial sample complexity for any desired accuracy.

Although uniform-PAC provides a powerful framework to unify regret 2 and PAC bound, it still fails to capture the instantaneous performance of the learning algorithm. In particular, a uniform-PAC algorithm could play a bad policy in some late but finite round t , even if it enjoys a good cumulative performance. This drawback impedes the application of uniform-PAC algorithms into high-stakes fields. Clinical trials, for example, place high demands on instantaneous performance for every treatment test, since patients need to be assigned with increasingly better treatments when more experimental data are available [Villar et al., 2015]. Hence, two natural questions arise:

Can we find a new metric that characterizes not only the cumulative performance but also the instantaneous performance?

If such a metric exists, is it optimally achievable by some algorithm?

In this paper, we answer both questions affirmatively. Our main contributions are summarized as follows.

We introduce a new metric called uniform last-iterate (ULI), which simultaneously characterizes cumulative and instantaneous performance of sequential decision-making algorithms. On one hand, ULI can characterize the instantaneous performance: the per-round suboptimality of any algorithm with ULI guarantee is upper bounded by a function, monotonically decreasing for late time t . On the other hand, we show that any algorithm with a (near-optimal) ULI guarantee is also (near-optimally) uniform-PAC, demonstrating that ULI can imply cumulative performance.

To answer the question whether ULI is achievable, we examine three common types of bandit algorithms in the finite arm setting. First, we provide a stronger analysis to show that many existing elimination-based algorithms indeed enjoy a near-optimal ULI guarantee. Then, we propose a metaalgorithm that enables any high-probability adversarial bandit algorithms, with a mild condition, to achieve a near-optimal ULI guarantee, and we show that such condition naturally holds for many adversarial bandit algorithms. Finally, we provide a hardness result showing that optimistic algorithms (e.g., lil'UCB [Jamieson et al., 2014]) cannot achieve near-optimal ULI guarantee. As lil'UCB is near-optimally uniform-PAC, our hardness result also implies that ULI is strictly stronger than uniform-PAC.

For linear bandits with infinitely-many arms, we propose an oracle-efficient 3 linear bandit algorithm with the ULI guarantee (with access to an optimization oracle). In particular, we propose an adaptive barycentric spanner technique, selecting finitely many base arms that can linearly represent all (possibly infinitely many) well-behaved arms. This technique generalizes the one in [Awerbuch and Kleinberg, 2008] for elimination-based algorithms by adaptively identifying spaces that active arms span. Leveraging the phased elimination algorithm [Lattimore et al., 2020], our algorithm can conduct the elimination over all arms by only playing a finite subset of arms and querying a linearly-constrained optimization oracle for only a polynomial number of times.

Finally, we propose a new algorithm for tabular episodic Markov decision processes (MDPs), which achieves a near-optimal ULI guarantee. In particular, our algorithm adapts uncertaintydriven reward functions to encourage exploration of the transition model, which ensures accurate estimations of value functions across all policies. The final ULI guarantee is achieved by conducting policy elimination.

Related work. In online decision-making problems, regret and PAC bounds are widely adopted to evaluate the cumulative performance of algorithms. More concretely, one line of research [Auer et al., 2002a,b, Abbasi-Yadkori et al., 2011, Li et al., 2010, Jin et al., 2018] aims to minimize the regret which measures the difference between the cumulative rewards of the selected policies and that of the

best policy in hindsight. The PAC guarantees are more common than the regret when studying the pure-exploration/best policy identification problems [Even-Dar et al., 2006, Kalyanakrishnan et al., 2012, Wagenmaker et al., 2022]. One of the popular PAC measures is ( δ, ϵ ) -PAC which suggests that with probability at least 1 -δ , the algorithm can output a near-optimal policy at most ϵ away from the optimal one by using a sample complexity polynomial in 1 /ϵ . Later, Dann et al. [2017] introduce a new framework called uniform-PAC to unify both metrics and develop a uniform-PAC algorithm for episodic Markov decision processes (MDPs). Subsequent works design uniform-PAC algorithms for MDPs with linear function approximation [He et al., 2021] and bounded Eluder dimension [Wu et al., 2023]. Though uniform-PAC strengthens regret and ( δ, ϵ ) -PAC bound, it still fails to characterize the instantaneous performance of online algorithms, i.e., a uniform-PAC algorithm, even with a good cumulative performance, can play bad policies for some late rounds.

A seemingly related performance measure is last-iterate convergence (LIC) which has been studied for optimizing MDPs [Moskovitz et al., 2023, Ding et al., 2023] and they use the primal-dual approach to formulate the problem of identifying an optimal policy in the constrained MDPs from a gametheoretic perspective. These works often require additional knowledge of the value functions and the LIC does not characterize the unknown dynamics of the environment. However, in our problem, the dynamics need to be learned as the algorithm sequentially interacts with the environment.

2 Preliminaries

2.1 Framework

We consider a general online sequential decision-making framework where a learner interacts with an environment with a fixed decision set. At each round t ∈ N , the learner makes a decision from the set and observes the corresponding reward(s). In what follows, we instantiate this framework to multi-armed bandits, linear bandits, and tabular episodic Markov decision processes (MDPs).

Multi-armed bandits. In the stochastic MAB setting, the arm (decision) set follows that A = [ K ] ≜ { 1 , . . . , K } . Each arm a ∈ [ K ] is associated with a fixed and unknown [0 , 1] -bounded distribution 4 such that ∀ t , reward X t,a is an i.i.d. sample from this distribution with mean µ a = E [ X t,a ] . Let A t be the arm played at round t and ∆ a = µ ⋆ -µ a be the suboptimality gap where µ ⋆ = max a ∈ [ K ] µ a .

Linear bandits. In the stochastic linear bandits setup, we assume that the arm set A ⊆ R d is compact. The reward of played arm A t at round t follows that X t,A t = ⟨ θ, A t ⟩ + η t where θ ∈ R d is a fixed but unknown parameter, and η t is conditionally 1 -subgaussian. Let ∆ a = sup b ∈A ⟨ θ, b -a ⟩ . We follow standard assumptions that ∥ θ ∥ 2 ≤ 1 , ∥ a ∥ 2 ≤ 1 for all a ∈ A , and ∆ a ≤ 1 for all a ∈ A .

Tabular episodic MDPs. A tabular episodic MDP is formalized as M = ( S , A , H, r, P, µ ) where S , A are finite state and action spaces with |S| = S, |A| = A , H is the horizon length, r = { r h } H h =1 where r h : S × A → [0 , 1] is a known reward function, { P h } h ∈ [ H ] where P h : S × A → ∆( S ) is a transition function, and µ is the initial state distribution. At the beginning of each episode t , the learner executes a policy π t = { π t,h : S → ∆( A ) } H h =1 . Then, starting from the initial state s t, 1 ∼ µ , for each stage h ∈ [ H ] , the learner repeatedly takes an action a t,h ∼ π t,h ( s t,h ) , observes reward r h ( s t,h , a t,h ) , and transits to the next state s t +1 ,h ∼ P h ( ·| s t,h , a t,h ) .

For any policy π and stage h , we define action value function Q π h ( s, a ) and value function V π h ( s ) as

The optimal action value function and value function at each stage h are denoted by V ⋆ h ( s ) = max π V π h ( s ) , and Q ⋆ h ( s, a ) = max π Q π h ( s, a ) respectively. Let ∆ π = E s 1 ∼ µ [ V ⋆ 1 ( s 1 ) -V π 1 ( s 1 )] .

Suboptimality notations. For MAB and linear bandits settings, the instantaneous suboptimality is ∆ t = ∆ A t , and for episodic MDP, ∆ t = ∆ π t . We use ∆ = inf a ∈ Π:∆ a > 0 ∆ a to denote the minimum suboptimality gap. Notice that it is possible that ∆ = 0 when, for example, arm set A is a ball.

2.2 Limitations of Existing Metrics

Regret and ( δ, ϵ ) -PAC are widely adopted to measure the performance. The regret is defined as:

Definition 2.1 (Regret) . For each fixed T ∈ N , the regret R T is defined as R T = ∑ T t =1 ∆ t .

Let N ϵ = ∑ ∞ t =1 I { ∆ t > ϵ } be the number of plays of policies whose suboptimality gap is greater than ϵ . The definition of ( δ, ϵ ) -PAC is given as:

Definition 2.2 ( ( δ, ϵ ) -PAC) . For any fixed δ, ϵ ∈ (0 , 1) , an algorithm is ( δ, ϵ ) -PAC (w.r.t. function F PAC) if there exists a function F PAC ( δ, ϵ ) polynomial in log( δ -1 ) and ϵ -1 such that

As shown by Dann et al. [2017], both regret and ( δ, ϵ ) -PAC have limitations. Specifically, an algorithm with sublinear regret bound may play suboptimal policies infinitely often. For the algorithm with ( δ, ϵ ) -PAC guarantee, it may not converge to the optimal policy when feeding the algorithm with more samples. Therefore, such an algorithm would play those policies with suboptimality gap, e.g., ϵ/ 2 infinitely often. Motivated by these limitations, Dann et al. [2017] introduce uniform-PAC as follows.

Definition 2.3 (Uniform-PAC) . An algorithm is uniform-PAC for some fixed δ ∈ (0 , 1) if there exists a function F UPAC ( δ, ϵ ) polynomial in log(1 /δ ) and ϵ -1 , such that

We also call F UPAC ( δ, ϵ ) the sample complexity function. Uniform-PAC is a stronger metric than regret and ( δ, ϵ ) -PAC since it leads to the following implications.

Theorem 2.4 (Theorem 3 in [Dann et al., 2017]) . If an algorithm is uniform-PAC for some δ with function F UPAC ( δ, ϵ ) = ˜ O ( α 1 /ϵ + α 2 /ϵ 2 ) 5 , where α 1 , α 2 > 0 are constant in ϵ and depend on log(1 /δ ) and K for MAB, d for linear bandits, and S, A, H for MDPs then, the algorithm guarantees:

P (lim t → + ∞ ∆ t = 0) ≥ 1 -δ ;

( δ, ϵ ) -PAC with F PAC ( δ, ϵ ) = F UPAC ( δ, ϵ ) for all ϵ > 0 ;

With probability at least 1 -δ , for all T ∈ N , R T = ˜ O ( √ α 2 T + α 1 + α 2 ) .

Limitations of Uniform-PAC. According to Theorem 2.4, uniform-PAC can imply a long-term convergence (the first bullet) and good cumulative performance (the second and the third bullets), but it does not capture the convergence rate of ∆ t for each round t . In other words, even if an algorithm enjoys uniform-PAC, it could still play a significantly bad policy for some very large but finite t . This would lead to catastrophic consequences in safety-critical applications.

2.3 New Metric: Uniform Last-Iterate Guarantee

To address the aforementioned issue, we introduce a new metric, formally defined below.

Definition 2.5 (Uniform last-iterate (ULI)) . An algorithm is ULI for some δ ∈ (0 , 1) if there exists a positive-valued function F ULI ( · , · ) , such that

where F ULI ( δ, t ) is polynomial in log(1 /δ ) and proportional to the product of power functions of log t and 1 /t (e.g., F ULI ( δ, t ) = polylog (1 /δ )(log t ) κ 1 t -κ 2 for some κ 1 , κ 2 ≥ 0 ).

According to Definition 2.5, the instantaneous suboptimality of any algorithm with the ULI guarantee can be bounded by a function F ULI ( δ, t ) . Moreover, F ULI ( δ, t ) decreases monotonically for large t if its power on 1 /t is strictly positive, which captures the convergence rate of ∆ t .

Note that the convergence rate of ∆ t is mainly determined by the power on 1 /t in F ULI. Moreover, as we will show shortly, an algorithm with the ULI guarantee automatically has a small regret bound. We therefore have the following lower bound on F ULI.

Theorem 2.6. For any bandit algorithm that achieves ULI guarantee for some δ with function F ULI ( δ, t ) , there exists a MAB instance such that F ULI ( δ, t ) = Ω ( t -1 / 2 ) .

We provide the proof in Appendix A. In the rest of the paper, we say an algorithm is near-optimal ULI if it achieves the ULI guarantee with F ULI ( δ, t ) = ˜ O (1 / √ t ) .

Then, we present the following theorem to show that ULI directly leads to uniform-PAC, implying that ULI also characterizes the cumulative performance of bandit algorithms.

Theorem 2.7. Suppose an algorithm achieves the ULI guarantee for some δ with function F ULI ( δ, t ) = polylog ( t/δ ) · t -κ where κ ∈ (0 , 1) . Then, we have,

the algorithm is uniform-PAC with function F UPAC ( δ, ϵ ) = O ( ϵ -1 κ · polylog ( δ -1 ϵ -1 ) ) .

with probability at least 1 -δ , ∀ T ∈ N , the regret R T is bounded by

when the minimum suboptimality gap of the input instance ∆ satisfies ∆ > 0 .

According to Theorem 2.7, if an algorithm is with near-optimal ULI guarantee (i.e., κ = 1 2 ), then it implies the near-optimality for uniform-PAC bound (the first bullet point) and anytime sublinear high-probability regret bound (the second bullet point). On the other hand, an algorithm with nearoptimal uniform-PAC bound does not necessarily enjoy a near-optimal ULI guarantee as shown in Section 3.2. The proof of Theorem 2.7 can be found in Appendix B.

Remark 2.8. Although a near-optimal ULI guarantee implies an anytime sublinear high-probability regret bound, it cannot give an anytime sublinear expected regret bound. This is because any algorithm with ULI guarantee is also uniform-PAC, but [Dann et al., 2017, Theorem 1] implies that no algorithm can be uniform-PAC and achieve anytime sublinear expected regret bound simultaneously.

3 Achieving Near-Optimal ULI in Bandits with Finite Arm-Space

In this section, we answer the question whether ULI is achievable for bandit problems. To this end, we examine three common types of bandit algorithms, including elimination-based algorithms, optimistic algorithms, and high-probability adversarial algorithms, in the finite arm setting, i.e., |A| = K .

3.1 Elimination Framework Achieving Near-Optimal ULI Guarantee

To examine whether the elimination-type algorithms can achieve the ULI guarantee, we first provide an elimination framework in Algorithm 1 that ensures the ULI guarantee, and we then show that most elimination-based algorithms fall into this framework. The following result shows that with a proper function f and a positive constant β , such an elimination framework ensures the ULI guarantee.

Theorem 3.1. For any given δ ∈ (0 , 1) , if there exists function f ( δ, t ) = t -κ polylog ( t/δ ) for some κ ∈ (0 , 1) and ∃ β > 0 , such that with probability 1 -δ , Eq. (1) holds for all t , then algorithm is ULI with F ULI ( δ, t ) = O ( f ( δ, t )) .

Algorithm 1 Elimination framework for ULI

Input : δ ∈ (0 , 1) , set A , function f ( · , · ) , and constant β . Initialize : active arm set A 0 = A . for t = 1 , 2 , . . . do Select an active set A t based on available observations as ( a ∗ is one of optimal arms) A t ⊆ { a ∈ A t -1 : ∆ a ≤ β · f ( δ, t ) } ∪ { a ⋆ } . (1) Play an arm A t ∈ A t and observe reward X t,A t .

Theorem 3.1 suggests that Eq. (1) is a sufficient condition for elimination-based algorithms to achieve the ULI guarantee. Now, we show that existing elimination algorithms indeed fall into such a framework. We here consider successive elimination (SE) and phased elimination (PE). Notice that we consider SE only for the MAB setting (called SE-MAB, e.g., Algorithm 3 in [Even-Dar et al., 2006]) but PE for both the MAB setting (called PE-MAB, e.g., exercise 6.8 in [Lattimore and Szepesvári, 2020]) and the linear bandit setting (called PE-L, e.g., Algorithm 12 of Chapter 22 in [Lattimore and Szepesvári, 2020]). Since those algorithms are standard, we defer their pseudocodes

to Appendix D. Given Theorem 3.1, the following results show that the elimination framework can be instantiated by these algorithms with proper functions and therefore they achieve the ULI.

Theorem 3.2. For any fixed δ ∈ (0 , 1) , elimination framework in Algorithm 1 can be instantiated by

SE-MAB for MAB to achieve the ULI with F ULI ( δ, t ) = O ( t -1 2 √ K log( δ -1 Kt ) ) .

PE-MAB for MAB to achieve the ULI with F ULI ( δ, t ) = O ( t -1 2 √ K log( δ -1 K log( t +1)) ) .

PE-L for linear bandits to achieve ULI with F ULI ( δ, t ) = O ( t -1 2 √ d log( δ -1 K log( t +1)log d ) ) .

Achieving ULI by adversarial bandit algorithms. Traditional elimination-based algorithms, including the ones mentioned above, typically require a carefully designed exploration strategy which is non-trivial even for linear bandits. Here, we provide an alternative way to achieve ULI by employing adversarial bandit algorithms to explore and then conduct the elimination. As shown in Appendix F, all adversarial bandit algorithms for both MAB and linear bandits that meet a certain condition can naturally be used to achieve the ULI guarantees similar to those of traditional elimination-based algorithms.

3.2 Lower Bound for Optimistic Algorithms

In this section, we present a lower bound to show that optimistic algorithms cannot achieve nearoptimal ULI guarantee. The procedure of optimistic algorithms is summarized as follows. After playing each arm once, at each round t , the algorithm plays an arm A t that satisfies

where N a ( t ) is the number of plays of arm a before round t , ̂ µ a ( N a ( t )) = ∑ t -1 s =1 X s,As I { A s = a } N a ( t ) is the empirical mean of arm a after N a ( t ) times play, and U δ ( N a ( t )) is a positive bonus function which encourages the exploration.

Wefirst consider optimistic algorithms e.g., upper confidence bound (UCB) [Lattimore and Szepesvári, 2020, Algorithm 3, Chapter 7], which enjoy (near)-optimal regret bounds. This type of algorithms typically uses the bonus function in the form of √ log(2 t 2 /δ ) /N a ( t ) 6 . However, the log t term forces the algorithm to play suboptimal arms infinitely often, and thus they cannot achieve the ULI guarantee (refer to Appendix E.3 for a detailed proof). Similarly, other variants [Audibert and Bubeck, 2010, Degenne and Perchet, 2016] with log t term in bonus function, also cannot achieve the ULI guarantee.

We then consider another optimistic-type algorithm, lil'UCB [Jamieson et al., 2014] which obtains the order-optimal instance-dependent sample complexity and avoids log( t ) term in U δ ( N a ( t )) . The bonus function of lil'UCB is as √ log ( δ -1 log + ( N a ( t )) ) /N a ( t ) where log + ( x ) = log (max { x, e } ) . Our main result for lil'UCB is presented as follows. The full analysis of lil'UCB is deferred to Appendix E.

Theorem 3.3. There exists a constant α ∈ (0 , 1) that for all ∆ ∈ (0 , α ) , running lil'UCB on the two-armed bandit instance with deterministic rewards and arm gap ∆ gives ∃ t = Ω ( ∆ -2 ) such that

Theorem 3.3 shows that lil'UCB is not near-optimal ULI, but it is unclear whether it can achieve the ULI guarantee. Recall from Theorem 3.2 that the elimination-based algorithms ensure that with high probability, ∀ t , ∆ t = ˜ O ( t -1 / 2 ) . Theorem 3.3 suggests that the convergence rate of lil'UCB is strictly worse than that of elimination-based algorithms, even if it enjoys a near-optimal cumulative performance [Jamieson et al., 2014]. In fact, this lower bound holds for all optimistic algorithms as long as the bonus function is in a similar form.

Remark 3.4 ( ULI is strictly stronger than uniform-PAC ) . For lil'UCB, the number of times playing suboptimal arms is finite with an order-optimal instance-dependent sample complexity, which implies that lil'UCB is near-optimal uniform-PAC. Therefore, Theorem 3.3 also shows that an algorithm with near-optimal uniform-PAC does not necessarily enjoy near-optimal ULI guarantee.

Algorithm 2 PE with adaptive barycentric spanner

Input : Compact arm set A , confidence δ ∈ (0 , 1) , and constant C > 1 .

1 for m = 1 , 2 , . . . do

2 Set T m = 256 C 4 · d 3 4 -m log ( δ -1 d 3 4 m ) .

3 Invoke Algorithm 12 with ( A , m, B m -1 , T m , C, ̂ θ m ) to find a C -approximate barycentric spanner B m for active arm set A m where A m is in Eq. (3).

4 Set π m ( a ) = 1 d for each a ∈ B m .

5 Play each arm a ∈ B m for n m ( a ) = ⌈ T m π m ( a ) ⌉ times.

6 Compute V m = I + ∑ a ∈B m n m ( a ) aa ⊤ and ̂ θ m +1 = V -1 m ∑ t ∈T m A t X t,A t where T m is a set that contains all rounds in phase m .

4 Achieving Near-Optimal ULI for Linear Bandits in Large Arm-Space

In this section, we propose a linear bandit algorithm that can handle the infinite number of arms. The compact arm set A is assumed to span R d and d is known.

4.1 Main Algorithm and Main Results

The starting point of our algorithm design is the phased elimination (PE) algorithm [Lattimore et al., 2020, Algorithm 12, Chapter 22]. However, PE in general is not feasible when the arm space is large (e.g., a continuous space). In this section, we present a carefully-designed algorithm to address the new challenges from large arm-spaces.

Issues of PE for large arm-space. PE needs to (i) compute (approximately) G -optimal design whose complexity scales linearly with |A| and (ii) compare the empirical mean of each arm, both of which are impossible when the arm set is infinite, e.g., A is a ball. A natural idea is to discretize A , e.g., constructing a ϵ -net, but the computational complexity has an exponential dependence on d , and the optimal arm does not necessarily lie in the net, which prevents the convergence to the optimal arm.

High-level idea behind our solution. To address the aforementioned issues, we propose an oracleefficient linear bandit algorithm in Algorithm 2 which can eliminate bad arms by efficiently querying an optimization oracle. Our algorithm equips PE with a newly-developed adaptive barycentric spanner technique. The proposed technique selects a finite representative arm set to represent (possibly infinite) active arm set and adaptively adjusts the selection of arms across phases. By conducting the (approximate) G -optimal design [Kiefer and Wolfowitz, 1960] on the representative arm set and playing each arm in the set according to the design, the algorithm can acquire accurate estimations uniformly over all active arms. Moreover, the adaptive barycentric spanner approach can be implemented by efficiently querying an optimization oracle in polynomial times.

The definition of barycentric spanner [Awerbuch and Kleinberg, 2008] is presented as follows.

Definition 4.1 ( C -approximate barycentric spanner) . Let A ⊆ R d be a compact set. The set B = [ b 1 , . . . , b d ] ⊆ A is a C -approximate barycentric spanner for A if each a ∈ A can be expressed as a linear combination of points in B with coefficients in the range of [ -C, C ] .

Why adaptive barycentric spanner? The primary reason that the non-adaptive barycentric spanner technique [Awerbuch and Kleinberg, 2008] fails to work for PE is that it requires the knowledge of the space that the active arm set spans. However, acquiring such knowledge is often difficult because the active arm set, which may span a proper linear subspace of R d , varies for different phases and the number of active arms could be infinite. Our algorithm shown in Algorithm 12 (whose pseudocode is deferred to Appendix G.2 due to space limit) can identify a barycentric spanner for active arm set adaptively for each phase, even if they do not span R d .

Algorithm procedure. Our algorithm proceeds in phases m = 1 , 2 , . . . , and each phase consists of consecutive rounds. At the beginning of phase m , the algorithm invokes subroutine Algorithm 12 to identify a C -approximate barycentric spanner B m which can linearly represent all arms in active arm set A m where A 1 = A and ∀ m ≥ 2

Algorithm 3 Tabular Episodic MDPs with ULI guarantee

where a ⋆ m is the empirical best arm and ̂ θ m is the estimation of unknown parameter θ .

Then, the algorithm assigns π m ( a ) = 1 d for each a ∈ B m . In fact, if we add ∪ i ∈I m e i √ T m (refer Appendix G for e i and I m ) back to B m and denote the new set by B ′ m , then π m : B ′ m → 1 d forms an optimal design. It is noteworthy that our algorithm only plays arms in B m as these added elements do not necessarily exist in the arm set A m . As each added element e i √ T m is close to zero, even if we only play arms in B m , we can still acquire accurate estimations uniformly over A m (refer to Appendix G.6 for details):

where V m = I + ∑ a ∈B m n m ( a ) aa ⊤ is the least squares matrix, used to estimate θ .

According to the standard analysis of linear bandit algorithms, the estimation error of ⟨ a, θ ⟩ is proportional to ∥ a ∥ V -1 m . Hence, the estimation errors of {⟨ a, θ ⟩} a ∈A m can be uniformly bounded by ˜ O ( Cd/ √ T m -1 ) , which is known to the learner. Finally, after playing each arm a ∈ B m for n m ( a ) times, the algorithm updates the empirical estimates V m and ̂ θ m +1 , and then steps into the next phase.

The main results of Algorithm 2 for achieving the ULI guarantee and computational complexity are given as follows. The full proof can be found in Appendix G.

Theorem 4.2. For any fixed δ ∈ (0 , 1) , Algorithm 2 achieves the ULI guarantee with function F ULI ( δ, t ) = O ( t -1 2 √ d 3 log( δ -1 dt ) ) . Moreover, in each phase, the number of calls to the optimization oracle given in Definition G.2 is O ( d 3 log C d ) .

Theorem 4.2 and Theorem 2.7 jointly suggest ˜ O ( d 3 / 2 √ T ) worst-case regret bound for the infinitearmed setting, which matches those of [Dani et al., 2008, Agrawal and Goyal, 2013, Hanna et al., 2023]. Compared with the lower bound Ω( d √ T ) given by Dani et al. [2008], our regret bound suffers an extra √ d factor, caused by the spanner technique. Yet, it remains open to find a computationally efficient linear bandit algorithm that can handle the infinite arm setting with general compact A and matches the Ω( d √ T ) lower bound.

Theorem 4.2 also shows that, for each phase, the spanner can be constructed by calling the oracle for only polynomial times. Compared to the computational efficiency of the algorithm in [Awerbuch and Kleinberg, 2008], the efficiency of our algorithm is only d times worse than theirs.

5 Achieving Near-Optimal ULI in Tabular Episodic MDPs

In this section, we propose a novel algorithm that achieves a near-optimal ULI guarantee in tabular episodic MDPs setup. The algorithm is formally presented in Algorithm 3.

High-level idea. Algorithm 3 conducts policy elimination over a policy set Π all which enumerates all deterministic policies. The key challenge here is to acquire accurate estimations of value functions uniformly for all deterministic policies. Note that naïvely playing all deterministic policies will incur linear dependence on | Π all | = A SH in F ULI, which is exponentially large. Hence, the algorithm invokes subroutine Algorithm 4 to exhaustively explore the environment, which ensures accurate estimations of the transition model and only suffers logarithmic dependence on | Π all | . Once the transition model can be well-approximated, the algorithm constructs an accurate estimation of the value function for every policy and then decides which policies should be eliminated.

Algorithm 4 Uniform estimation for value functions

Input : confidence δ ∈ (0 , 1) , policy set Π ⊆ Π all , duration T . Initialize : randomly pick a policy π 1 ∈ Π , N 1 ,h ( s, a ) = N 1 ,h ( s, a, s ′ ) = 0 for all ( h, s, a, s ′ ) .

for t = 1 , . . . , T do Observe initial state s t, 1 ∼ µ . for h = 1 , . . . , H do Take action a t,h = π t,h ( s t,h ) and observe s t,h +1 ∼ P h ( ·| s t,h , a t,h ) . Increase counters N t,h ( s t,h , a t,h , s t,h +1 ) + ← 1 and N t,h ( s t,h , a t,h ) + ← 1 . Update estimates ̂ P t,h ( s ′ | s, a ) = N t,h ( s,a,s ′ ) max { 1 ,N t,h ( s,a ) } for all ( s, a, s ′ ) ∈ S × A × S . Update bonus function b t = { b t,h } h ∈ [ H ] where b t,h ( · , · ) is updated according to Eq. (5). Get { ̂ V π t, 1 ( s 1 ) } π ∈ Π by invoking Algorithm 5 with input (Π , b t /H, { ̂ P T,h } h ∈ [ H ] , b t , s t, 1 ) . Update policy π t +1 = argmax π ∈ Π ̂ V π t, 1 ( s 1 ) . .

Algorithm 5 Construct estimated value function

Input : policy set Π ⊆ Π all , reward function r , transition ̂ P , bonus function b , initial state s 1 . for π ∈ Π do

Output : { ̂ V π 1 ( s 1 ) } π ∈ Π .

More concretely, Algorithm 3 first accepts a set of all deterministic policies and then it proceeds in phases m = 1 , 2 , . . . . In each phase m , subroutine Algorithm 4 is invoked to learn the transition model. Specifically, the subroutine inherits the structure of UCB-VI [Azar et al., 2017], but more importantly the algorithm pretends to be agnostic to the reward function and uses uncertainty-driven reward functions { b t,h ( s, a ) /H } t,h where

where N t,h ( s, a ) is the number of times of visiting ( s, a ) at stage h up to episode t . Note that Eq. (5) captures the uncertainty of visitation a state-action pair, i.e., more visitations of ( s, a ) , larger N t,h ( s, a ) , and less uncertainty. This modification, inspired by [Wang et al., 2020] forces the algorithm to aggressively explore the environment, in the sense that the algorithm prefers to play a policy that maximizes the uncertainty.

The following theorem shows that our algorithm achieves the ULI guarantee.

Theorem 5.1. For any fixed δ ∈ (0 , 1) , Algorithm 3 achieves the ULI guarantee with function F ULI ( δ, t ) = O ( t -1 2 √ S 3 AH 5 log ( tSAH/δ ) ) .

Theorem 5.1 shows near-optimality of Algorithm 3 w.r.t. episode t . Unfortunately, the regret bound implied by our ULI result incurs suboptimal dependence on S, H , and the logarithmic term, and our algorithm is not computational efficient. We leave these improvements for future work.

6 Conclusions and Future Work

In this paper, we propose a new metric, the uniform last-iterate (ULI) guarantee, which captures both instantaneous and cumulative performance of sequential decision-making algorithms. To answer

whether ULI is (optimally) achievable, we first examine three types of bandit algorithms in the finitearm setting. Specifically, we provide stronger analysis to show that elimination-based algorithms naturally achieve near-optimal ULI guarantees. We also provide a reduction-based approach to enable any high-probability adversarial algorithms, with a mild condition, to achieve near-optimal ULI guarantees. We further provide a negative result for optimistic bandit algorithms showing that they cannot achieve near-optimal ULI guarantee. Furthermore, in the large arm space setting, we propose an oracle-efficient linear bandit algorithm, equipped with the novel adaptive barycentric spanner technique. Finally, we propose a new algorithm, which adapts uncertainty-driven reward functions into policy elimination to achieve a ULI guarantee in tabular episodic MDPs.

Some natural directions are to improve our current results, including proposing a ULI algorithm for linear bandits with infinite arms and proposing a computationally efficient algorithm, possibly based on the action-elimination approach, for tabular MDPs. We also summarize other interesting directions as follows.

Design (computationally efficient) algorithms for MDPs with linear function approximation. The main challenge is to bypass any dependence on the number of states which is possibly infinite. Thus, generalizing our RL algorithm, which enumerates all deterministic policies, to linear MDPs does not work.

Design (computationally efficient) algorithms for Episodic MDPs with only logarithmic dependence on H (a.k.a. horizon-free). In our attempts, we use the doubling trick on ϵ in existing ( δ, ϵ ) -PAC horizon-free RL algorithms. In this case, we run the algorithm in phases with ϵ, ϵ/ 2 , ϵ/ 4 , . . . . The main difficulty is to leverage the information learned from the previous phase to guide the algorithm to play an improved policy at the next phase as required by ULI. Thus, we conjecture that there might exist fundamental barriers to simultaneously achieve ULI guarantee and a logarithmic dependence on H .

It could be also interesting to investigate some empirical issues when deploying ULI algorithms in high-stakes fields. Typically, optimistic algorithms initially incur a lower regret than ULI algorithms, but their regret will exceed those of ULI algorithms at a certain juncture. At this juncture, ULI algorithms have eliminated all suboptimal arms, whereas optimistic algorithms continue exploring as time evolves. Hence, identifying this turning point could be beneficial for deploying ULI algorithms in high-stakes domains.

References

Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems , 2011.

Agarwal, D. Hsu, S. Kale, J. Langford, L. Li, and R. Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning , 2014.

Agrawal and N. Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning , 2013.

J.-Y. Audibert and S. Bubeck. Regret bounds and minimax policies under partial monitoring. Journal of Machine Learning Research , 11(94):2785-2836, 2010.

J.-Y. Audibert, S. Bubeck, and R. Munos. Best arm identification in multi-armed bandits. In Conference on Learning Theory , 2010.

Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 2002a.

Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002b.

Awerbuch and R. Kleinberg. Online linear optimization and adaptive routing. Journal of Computer and System Sciences , 74(1):97-114, 2008.

G. Azar, I. Osband, and R. Munos. Minimax regret bounds for reinforcement learning. In Proceedings of the International Conference on Machine Learning , 2017.

L. Bartlett, V . Dani, T. P. Hayes, S. M. Kakade, A. Rakhlin, and A. Tewari. High-probability regret bounds for bandit online linear optimization. In Conference on Learning Theory , 2008.

Bibaut, A. Chambaz, and M. Laan. Generalized policy elimination: an efficient algorithm for nonparametric contextual bandits. In Conference on Uncertainty in Artificial Intelligence , 2020.

Chen, T. Lin, I. King, M. R. Lyu, and W. Chen. Combinatorial pure exploration of multi-armed bandits. In Advances in Neural Information Processing Systems , 2014.

Dani, T. P. Hayes, and S. M. Kakade. Stochastic linear optimization under bandit feedback. In Conference on Learning Theory , 2008.

Dann, T. Lattimore, and E. Brunskill. Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning. In Advances in Neural Information Processing Systems , 2017.

Degenne and V. Perchet. Anytime optimal algorithms in stochastic multi-armed bandits. In International Conference on Machine Learning , 2016.

Ding, C.-Y. Wei, K. Zhang, and A. Ribeiro. Last-iterate convergent policy gradient primal-dual methods for constrained mdps. arXiv preprint arXiv:2306.11700 , 2023.

Even-Dar, S. Mannor, Y. Mansour, and S. Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research , 7(6), 2006.

Hanna, L. Yang, and C. Fragouli. Efficient batched algorithm for contextual linear bandits with large action space via soft elimination. In Advances in Neural Information Processing Systems , 2023.

He, D. Zhou, and Q. Gu. Uniform-pac bounds for reinforcement learning with linear function approximation. In Advances in Neural Information Processing Systems , 2021.

Jamieson, M. Malloy, R. Nowak, and S. Bubeck. lil'ucb: An optimal exploration algorithm for multi-armed bandits. In Conference on Learning Theory , 2014.

Jin, Z. Allen-Zhu, S. Bubeck, and M. I. Jordan. Is q-learning provably efficient? In Proceedings of the International Conference on Neural Information Processing Systems , 2018.

Jin, T. Jin, H. Luo, S. Sra, and T. Yu. Learning adversarial markov decision processes with bandit feedback and unknown transition. In Proceedings of the International Conference on Machine Learning (ICML) , 2020.

Jin, J. Shi, X. Xiao, and E. Chen. Efficient pure exploration in adaptive round model. In Advances in Neural Information Processing Systems , 2019.

Kalyanakrishnan, A. Tewari, P. Auer, and P. Stone. Pac subset selection in stochastic multi-armed bandits. In International Conference on Machine Learning , 2012.

Kiefer and J. Wolfowitz. The equivalence of two extremum problems. Canadian Journal of Mathematics , 12:363-366, 1960.

Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Lattimore, C. Szepesvari, and G. Weisz. Learning with good feature representations in bandits and in rl with a generative model. In International Conference on Machine Learning , 2020.

C.-W. Lee, H. Luo, C.-Y. Wei, and M. Zhang. Bias no more: high-probability data-dependent regret bounds for adversarial bandits and mdps. In Advances in neural information processing systems , 2020.

C.-W. Lee, H. Luo, C.-Y. Wei, M. Zhang, and X. Zhang. Achieving near instance-optimality and minimax-optimality in stochastic and adversarial linear bandits simultaneously. In International Conference on Machine Learning , 2021.

Li, W. Chu, J. Langford, and R. E. Schapire. A contextual-bandit approach to personalized news article recommendation. In International Conference on World Wide Web , 2010.

Li, L. Ratliff, K. G. Jamieson, L. Jain, et al. Instance-optimal pac algorithms for contextual bandits. In Advances in Neural Information Processing Systems , 2022.

Moskovitz, B. O'Donoghue, V. Veeriah, S. Flennerhag, S. Singh, and T. Zahavy. Reload: Reinforcement learning with optimistic ascent-descent for last-iterate convergence in constrained mdps. In International Conference on Machine Learning , 2023.

Plevrakis and E. Hazan. Geometric exploration for online control. In Advances in Neural Information Processing Systems , 2020.

S. Villar, J. Bowden, and J. Wason. Multi-armed bandit models for the optimal design of clinical trials: benefits and challenges. Statistical science: a review journal of the Institute of Mathematical Statistics , 30(2):199, 2015.

J. Wagenmaker, M. Simchowitz, and K. Jamieson. Beyond no regret: Instance-dependent pac reinforcement learning. In Conference on Learning Theory , pages 358-418. PMLR, 2022.

Wang, S. S. Du, L. Yang, and R. R. Salakhutdinov. On reward-free reinforcement learning with linear function approximation. Advances in neural information processing systems , 2020.

Wu, J. He, and Q. Gu. Uniform-pac guarantees for model-based rl with bounded eluder dimension. In Conference on Uncertainty in Artificial Intelligence , 2023.

Zhu, D. J. Foster, J. Langford, and P. Mineiro. Contextual bandits with large action spaces: Made practical. In International Conference on Machine Learning , 2022.

Appendix

A Proof of Theorem 2.6

We prove this claim by contradiction. Suppose that there exists an algorithm that can achieve the ULI guarantee with function F ULI ( δ, t ) ≤ c 0 1 t 1 2 + α for some α ∈ (0 , 1 2 ) and constant c 0 > 0 which may depend on log(1 /δ ) and K for MAB or d for linear bandits. We consider a K -armed bandit instance ( K ≥ 2 ) with Gaussian rewards and all suboptimality gaps are bounded by one.

By the definition of ULI, we have that with probability at least 1 -δ , for all T ∈ N with T 1 2 -α > t 0 , we have

where the first inequality holds as we assume all suboptimal gaps are bounded by one, and ∆ t ≤ F ULI ( δ, t ) for all t ≥ t 0 .

Since 1 2 -α ∈ (0 , 1 2 ) , there will be a contradiction to the regret lower bound [Lattimore and Szepesvári, 2020, Corollary 17.3] for the Gaussian MAB setup.

B Proof of Theorem 2.7

Theorem B.1 (Restatement of Theorem 2.7) . Suppose an algorithm achieves the ULI guarantee for some δ with function F ULI ( δ, t ) = polylog ( t/δ ) · t -κ where κ ∈ (0 , 1) . Then, we have,

the algorithm is also uniform-PAC with

with probability at least 1 -δ , ∀ T ∈ N , the regret R T is bounded by

when the minimum suboptimality gap of the input instance ∆ satisfies ∆ > 0 .

B.1 Proof of the first bullet: ULI guarantee implies uniform-PAC bound

In this subsection, we show the first bullet of Theorem 2.7. We define the following events for F ULI = polylog ( t/δ ) · t -κ :

which indicates the occurrence the ULI, and denote the event B as:

For some δ ∈ (0 , 1) , if event A holds for an algorithm with probability at least 1 -δ , then the algorithm is with ULI guarantee. Therefore, to prove the claimed result, it suffices to show that conditioning on event A that for any algorithm enjoys ULI guarantee with a function F ULI ( δ, t ) , one can find a function F UPAC ( δ, ϵ ) such that event B also holds.

As for any fixed δ ∈ (0 , 1) , F ULI ( δ, t ) = polylog ( t/δ ) · t -κ is monotonically decreasing for large t , ∃ U ∈ R > 0 such that F ULI ( δ, t ) ≤ U for all t ∈ N and F ULI ( δ, t ) = U is attainable by some t . Then, we consider any given ϵ > 0 with two cases as follows.

Case 1: ϵ < U . Again, by the fact that F ULI ( δ, t ) = polylog ( t/δ ) · t -κ is monotonically decreasing for large t , there should exist a round t ∈ N such that F ULI ( δ, s ) ≤ ϵ for all s ≥ t . Let t 0 be the round with the maximal index such that F ULI ( δ, t 0 ) > ϵ , which gives

As a result, the number of times that ∆ t > ϵ occurs is at most

Then, one can find constants c 0 > 0 and z ∈ (0 , κ ) such that polylog ( t/δ ) ≤ c 0 · polylog (1 /δ ) t z for all t ∈ R ≥ 1 , thereby polylog ( t 0 /δ ) ≤ c 0 · polylog (1 /δ ) t z 0 . Combining polylog ( t 0 /δ ) ≤ c 0 · polylog (1 /δ ) t z 0 and F ULI ( δ, t 0 ) > ϵ gives

which immediately leads to

Then, we can further show that

where the third inequality follows that polylog ( t/δ ) is monotonically increasing for all t > 0 and Eq. (7).

Case 2: ϵ ≥ U . Conditioning on event A and by the definition of U , we have ∆ t ≤ F ULI ( δ, t ) ≤ U ≤ ϵ for all t ∈ N , which implies ∞ ∑ t =1 I { ∆ t > ϵ } = 0 . Therefore, any positive function works there and we still choose F UPAC ( δ, ϵ ) the same as above.

This argument holds for all ϵ > 0 . Therefore, if an algorithm satisfies the ULI guarantee, then P ( A ) ≥ 1 -δ , and with the above statement, we can find a function F UPAC such that P ( B ) ≥ 1 -δ , which concludes the proof.

B.2 Proof of the second bullet: ULI implies regret bounds

For some fixed δ ∈ (0 , 1) , let us consider an algorithm enjoys the ULI guarantee with function F ULI ( · , · ) . The following analysis will condition on event A given in Eq. (6).

Gap-dependent bound. Recall that ∆ > 0 is the minimum gap. We replace ϵ with ∆ / 2 in the proof of in Appendix B.1, case 1, and similarly define t 0 ∈ N as the maximal round such that F ULI ( δ, t 0 ) = polylog ( t 0 /δ ) · t -κ 0 > ∆ / 2 . Then, we apply a similar argument (in Appendix B.1,

case 1), which gives that that for some constants z ∈ (0 , κ ) and c 0 > 0 :

where the last inequality holds by taking the minimum of Eq. (7) and Eq. (8). Therefore, the algorithm will incur no regret, i.e., ∆ t = 0 for all t ≥ t 0 For every T ∈ N , the regret R T is bounded by

where the first inequality holds since we condition on the event A , the second inequality holds due to the definition of t 0 , the third inequality uses the fact that polylog ( t/δ ) is monotonically increasing for all t > 0 .

Then, one can further show

where the second inequality applies the second term of Eq. (9) and the last inequality uses the first term of Eq. (9).

Notice that these bounds in all three cases hold for all T ∈ N , and thus the proof of gap-dependent bound is complete.

Gap-independent bound. We have

C Proof of Theorem 3.1

̸

For shorthand, we use E to denote the event that Eq. (1) holds for all t ∈ N . From the assumption, E holds with probability at least 1 -δ . The following proof is straightforward by the definition of E . We conditions on E and consider an arbitrary round t . If A t = a ⋆ , the claim trivially holds as ∆ A t = 0 . We then consider A t = a ⋆ . Since Algorithm 1 pulls an arm A t ∈ A t at each t . From the definition of E , we have ∆ A t ≤ β · f ( δ, t ) . Therefore, Algorithm 1 achieves the ULI guarantee with F ULI ( δ, t ) = O ( f ( δ, t )) .

D Proof of Theorem 3.2

D.1 ULI guarantee for SE-MAB algorithm

Consider the successive elimination (SE-MAB) algorithm (e.g., Algorithm 3 in [Even-Dar et al., 2006]) in the MAB setting where A = [ K ] . We present SE-MAB algorithm in Algorithm 6 and define some notations. Let N a ( t ) be the number of times that arm a has been pulled before round t and let ̂ µ a ( s ) be the empirical mean of arm a after s ∈ N number of plays. We define the confidence width for each s ∈ N as

The SE-MAB algorithm always chooses the arm (from the current active arm set) with the minimum number of pulls. If there exist two arms a, j ∈ [ K ] such that N a ( t ) = N j ( t ) , then, the algorithm chooses the arm with the smallest index.

Algorithm 6 Successive elimination for multi-armed bandit (SE-MAB)

Definition D.1. Let E be the event that | ̂ µ a ( N a ( t )) -µ a | ≤ wd ( N a ( t )) holds for all t ∈ N and all a ∈ A t .

Proof. Following the standard trick (e.g., [Audibert et al., 2010]), we rewrite the empirical mean for every a, n as ̂ µ a ( n ) = 1 n ∑ n i =1 R i,a where R i,a is the reward of i -th pull of arm a . Let us fix both s ∈ N and a ∈ [ K ] . By Hoeffding's inequality, with probability at least 1 -δ ′

Choosing δ ′ = δ 2 Ks 2 and applying a union bound over a ∈ A t ( |A t | ≤ K , ∀ t ), we have with probability at least 1 -δ/ (2 s 2 ) ,

We apply a union bound over all s ∈ N and use the fact that ∑ ∞ s =1 δ 2 s 2 ≤ δ to finish the proof.

Lemma D.3. Suppose that E occurs where E is given in Definition D.1. For all t ∈ N , a ⋆ ∈ A t holds.

Proof. We prove this by induction. For t = 1 , a ⋆ ∈ A 1 trivially holds. Suppose that a ⋆ ∈ A t . To show a ⋆ ∈ A t , it suffices to show that at the end of round t , arm a ⋆ is deemed as a good arm. One can use the definition of E to show that for every a ∈ A t , the following holds.

Note that the second inequality above can hold since the inductive hypothesis gives that a ⋆ ∈ A t . The above inequality shows that a ⋆ will not be eliminated at the end of round t thereby still being active at round t +1 . Once the induction is done, the proof is complete.

Lemma D.4. Suppose that E occurs where E is given in Definition D.1. For all t ∈ N and all arms a ∈ A if a ∈ A t , then

Proof. Consider any round t ∈ N and any active arm a ∈ A t . For simplicity, we assume that the first arm is the unique optimal arm 7 . If a is an optimal arm, then ∆ a = 0 and we hold the claim trivially. Then, it suffices to consider an arm a ∈ A t with ∆ a > 0 . Notice that wd ( x ) is monotonically decreasing for all x ≥ 1 as long as K ≥ 2 . Thus, we find a minimum natural number T a ∈ N such that

With this definition, we have

Recall that N a ( t ) is the number of plays of arm a before round t . We first show N a ( t ) ≤ T a . As Algorithm 6 pulls arms in a round-robin fashion, if arm a is pulled for T a times, then, the optimal arm a ⋆ is also pulled for T a times, thereby T a = T a ⋆ . From Lemma D.3, optimal arm a ⋆ is active for all t ∈ N and thus we can use it as a comparator.

According to elimination rule, this inequality shows that arm a will be eliminated after T a pulling times, which suggests that N a ( t ) ≤ T a . Since the T a is the minimum natural number which holds Eq. (12), we use N a ( t ) ≤ T a to show

where the last inequality follows from the fact that N a ( t ) ≤ t . Rearranging the above, we have

Again, as Algorithm 6 pulls arms in a round-robin fashion, we have t ≤ K ( N a ( t ) + 1) and show

which gives ∆ a ≤ √ 14 K log(4 Kt 2 /δ ) t . Conditioning on event E the argument holds for all t ∈ N , which thus completes the proof.

Proof of Theorem 3.2 for SE-MAB. Once Lemma D.3 and Lemma D.4 hold, Theorem 3.1 gives that for any fixed δ ∈ (0 , 1) , SE-MAB achieves the ULI guarantee with a function

Therefore, the proof of Theorem 3.2 for SE-MAB is complete.

D.2 ULI guarantee for PE-MAB

In MAB setting with A = [ K ] , we further consider phased elimination algorithm (e.g., Exercise 6.8 of [Lattimore and Szepesvári, 2020]) shown in Algorithm 7 (called PE-MAB). The algorithm proceeds with phases ℓ = 1 , 2 , . . . , and each phase ℓ includes consecutive rounds, with an exponential increase. In phase ℓ , the algorithm sequentially pulls every arm a ∈ [ K ] for m ℓ times where

Once all arms are pulled m ℓ times, the algorithm steps into the next phase ℓ +1 . The counter N a ( ℓ ) records the number of times that arm a is pulled in phase ℓ and ̂ µ a ( m ℓ ) is the empirical mean by using m ℓ samples only from phase ℓ .

Algorithm 7 Phased elimination for multi-armed bandit (PE-MAB)

Input : confidence δ ∈ (0 , 1)

.

Initialize : active arm set

A 1 = [ K ] .

for ℓ = 1 , 2 , . . . do

Play every arm a ∈ A ℓ for m ℓ times and observe corresponding rewards. Update empirical means { ̂ µ a ( m ℓ ) } a ∈A ℓ only using samples from phase ℓ Update active arm set A ℓ +1 as

.

Definition D.5. Let E be the event that | ̂ µ a ( m ℓ ) -µ a | ≤ 2 -ℓ -1 holds for all ℓ ∈ N and all a ∈ A ℓ .

We first give the following lemmas, whose proof is similar to those of Lemma D.2 and Lemma D.3. Lemma D.6. P ( E ) ≥ 1 -δ .

Proof. We first fix both ℓ ∈ N and a ∈ [ K ] . By Hoeffding's inequality, with probability at most δ ′ , we have

Choosing δ ′ = δ 2 Kℓ 2 and applying union bounds over a ∈ A ℓ ( A ℓ is at most K ), with probability at most δ/ 2 ℓ 2 , we have

We apply union bounds over all ℓ ∈ N and use the fact that ∑ ∞ ℓ =1 δ 2 ℓ 2 ≤ δ to finish the proof.

Lemma D.7. Suppose that E occurs where E is given in Definition D.5. For all ℓ ∈ N , a ⋆ ∈ A ℓ .

Proof. We prove this by induction. For the base case ℓ = 1 , the claim trivially holds. Suppose the claim holds for ℓ , and then we will show that a ∈ A ℓ +1 . For all ℓ ∈ N and all a ∈ A ℓ , we have

where the second inequality holds because the induction hypothesis ensures a ⋆ ∈ A ℓ and thus we can use the definition of E . Based on the elimination rule, a ⋆ ∈ A ℓ +1 . Once the induction is done, the proof is complete.

Lemma D.8. Suppose that E occurs where E is given in Definition D.5. For each arm a with ∆ a > 0 , it will not be in A ℓ for all phases ℓ ≥ ℓ a +1 where ℓ a is the smallest phase such that ∆ a 2 > 2 -ℓ a .

Proof. Consider any arm a with ∆ a > 0 . We only need to consider a ∈ A ℓ a and otherwise, the claimed result holds trivially. Recall that ℓ a is the smallest phase such that ∆ a 2 > 2 -ℓ a and one can show

where the first inequality follows from Lemma D.7 that a ⋆ ∈ A ℓ a and the second inequality holds due to E .

According to the elimination rule, arm a will not be in phases ℓ for all ℓ ≥ ℓ a +1 .

Lemma D.9. Let ℓ ( t ) be the phase in which round t lies. Then, ℓ ( t ) ≤ log 2 ( t +1) for all t ∈ N .

Proof. We prove this by contradiction. Suppose that ∃ t ∈ N that ℓ ( t ) > log 2 ( t +1) . Note that we can further assume ℓ ( t ) ≥ 2 since one can easily verify that for all t such that ℓ ( t ) = 1 , ℓ ( t ) ≤ log 2 ( t +1) must hold. We have

where the third inequality bounds ℓ ( t ) in the logarithmic term by ℓ ( t ) ≥ 2 and bound the other ℓ ( t ) > log 2 ( t +1) by assumption. Therefore, once a contradiction occurs, the proof is complete.

Lemma D.10. Let ℓ ( t ) be the phase in which round t lies. Suppose that E occurs where E is given in Definition D.5. For all t ∈ N and all a ∈ [ K ] , if a ∈ A ℓ ( t ) , then

Proof. If a ∈ A ℓ ( t ) is optimal, then, ∆ a = 0 and the claim trivially holds. In what follows, we only consider arm a ∈ A ℓ ( t ) with ∆ a > 0 . From Lemma D.8, if an arm a ∈ A ℓ ( t ) is with ∆ a > 0 , then, ℓ ( t ) ≤ ℓ a where ℓ a is defined in Lemma D.8. Thus, the total number of rounds that such an arm a is active is at most

where the first inequality follows from ⌈ x ⌉ ≤ 2 x for all x ≥ 1 , the second inequality bounds ℓ ( t ) ≤ ℓ a , the fourth inequality follows from the definition of ℓ a , i.e., ∆ a 2 ≤ 2 -( ℓ a -1) , and the last inequality follows from Lemma D.9 that ℓ ( t ) ≤ log 2 ( t +1) .

Since this argument holds for each round t and each arm a ∈ A ℓ ( t ) conditioning on E , the proof is complete.

Proof of Theorem 3.2 for PE-MAB. Once Lemma D.7 and Lemma D.10 hold, Theorem 3.1 gives that for any fixed δ ∈ (0 , 1) , PE-MAB achieves the ULI guarantee with a function

Therefore, the proof of Theorem 3.2 for PE-MAB is complete.

D.3 ULI guarantee for PE-linear

Algorithm 8 Phased elimination for linear bandit (PE-linear)

Input : confidence δ ∈ (0 , 1) .

Initialize : active arm set A 1 = A .

Find a design π ℓ ∈ ∆( A ℓ ) with

where G ℓ = ∑ a ∈A ℓ π ℓ ( a ) aa T .

Play every arm a ∈ A ℓ for m ℓ ( a ) = ⌈ π ℓ ( a ) m ℓ ⌉ times and observe corresponding rewards.

Update the empirical estimate as

Update active arm set

We consider a phased elimination algorithm (e.g., algorithm in Chapter 22 of [Lattimore and Szepesvári, 2020]) for linear bandits setting with a finite arm set A = [ K ] . The algorithm proceeds with phases ℓ = 1 , 2 , . . . , and in each phase ℓ , the algorithm first computes a design π ℓ ∈ ∆( A ℓ ) over all active arms where ∆( A ℓ ) is the set of all Radon probability measures over set A ℓ . Rather than computing an exact design in [Lattimore and Szepesvári, 2020], we follow Lattimore et al. [2020] to compute a nearly-optimal design (13), which can be efficiently implemented. Then, PE-linear plays each arm a ∈ A ℓ for m ℓ ( a ) times and updates the active arm set by using the estimates in this phase.

Let us define T ℓ be a set that contains all rounds in phase ℓ and

Lemma D.11. For all ℓ ∈ N and all a ∈ A ℓ , ∥ a ∥ 2 V -1 ℓ ≤ 2 d m ℓ holds.

Proof. For each ℓ ∈ N and a ∈ A ℓ , one can show

where the first inequality follows from Eq. (15) and the fact that if we let A = m ℓ ∑ a ∈A ℓ π ℓ ( a ) aa ⊤ and B = V ℓ then, ∥ a ∥ A -1 ≥ ∥ a ∥ B -1 holds since A -1 ⪰ B -1 and the second inequality uses Eq. (13).

Definition D.12. Let E be the event that ∣ ∣ ∣ 〈 ̂ θ ℓ -θ, a 〉∣ ∣ ∣ ≤ 2 -ℓ holds for all ℓ ∈ N and all a ∈ A ℓ . Lemma D.13. P ( E ) ≥ 1 -δ .

Proof. Consider a fixed phase ℓ and a fixed arm a ∈ A ℓ . We start from the following decomposition.

By Eq. (20.2) of [Lattimore and Szepesvári, 2020], we have with probability at least 1 -δ/ (2 ℓ 2 K ) ,

where the last inequality applies Lemma D.11. Finally, applying union bounds over all a, ℓ completes the proof.

Lemma D.14. Suppose that E occurs where E is given in Definition D.12. We have a ⋆ ∈ A ℓ for all ℓ ∈ N .

Proof. This can be proved by using induction on ℓ via the same reasoning of Lemma D.3.

Lemma D.15. Suppose that E occurs where E is given in Definition D.12. For each arm a with ∆ a > 0 , it will not be in A ℓ for all phases ℓ ≥ ℓ a + 1 where ℓ a is the smallest phase such that ∆ a 4 > 2 -ℓ a .

Proof. Consider any arm a with ∆ a > 0 . Let ℓ a be the smallest phase such that ∆ a 4 > 2 -ℓ a and we have

where the first inequality follows from Lemma D.14 that a ⋆ ∈ A ℓ for all ℓ .

According to the elimination rule, arm a will not be in A ℓ for all phases ℓ ≥ ℓ a +1 . Since conditioning on E , this argument holds for every arm a , the proof is complete.

Lemma D.16. Let ℓ ( t ) be the phase in which round t lies. Then, ℓ ( t ) ≤ log 2 ( t +1) for all t ∈ N .

Proof. We prove this by contradiction. Suppose that ∃ t ∈ N that ℓ ( t ) > log 2 ( t +1) . Note that we can further assume ℓ ( t ) ≥ 2 since one can easily verify that for all t such that ℓ ( t ) = 1 , ℓ ( t ) ≤ log 2 ( t +1) must hold. We have

where the fourth inequality bounds ℓ ( t ) in the logarithmic term by ℓ ( t ) ≥ 2 and bound the other ℓ ( t ) > log 2 ( t +1) by assumption. Therefore, once a contradiction occurs, the proof is complete.

Lemma D.17. Let ℓ ( t ) be the phase in which round t lies. Suppose that E occurs where E is given in Definition D.12. For all t ∈ N and all a ∈ [ K ] , if a ∈ A ℓ ( t ) , then

Proof. If a ∈ A ℓ ( t ) is optimal, then, ∆ a = 0 and the claim trivially holds. In what follows, we only consider arm a ∈ A ℓ ( t ) with ∆ a > 0 . From Lemma D.15, if an arm a ∈ A ℓ ( t ) is with ∆ a > 0 , then, ℓ ( t ) ≤ ℓ a where ℓ a is defined in Lemma D.15. Then, the total number of rounds that such an arm a is active is at most

̸

̸

where the first inequality applies ⌈ π s ( a ) m s ⌉ ≤ π s ( a ) m s +1 and Eq. (13), the second inequality uses the definition of m s (see Eq. (15)), the fourth inequality follows from ℓ ( t ) ≤ ℓ a and Lemma D.16 gives ℓ ( t ) ≤ log 2 ( t +1) , and the last inequality uses ∆ a 4 ≤ 2 -( ℓ a -1) to apply ℓ a ≤ log 2 ( 8 / ∆ a ) .

Since this argument holds for each round t and each arm a ∈ A ℓ ( t ) conditioning on E , the proof is complete.

Proof of Theorem 3.2 for PE-linear. Once Lemma D.14 and Lemma D.17 hold, Theorem 3.1 gives that for any fixed δ ∈ (0 , 1) , PE-linear achieves the ULI guarantee with a function

Therefore, the proof of Theorem 3.2 for PE-linear is complete.

E Omitted Details of Section 3.2

The bonus function that we here consider for lil'UCB is as:

The choice of U δ ( x ) in Eq. (16) is slightly different from that of [Jamieson et al., 2014] because we consider for any δ ∈ (0 , 1) and they constrain the choice of δ in a more restricted range. The choice of U δ ( x ) is motivated from another lemma of law of iterated logarithm, given in [Dann et al., 2017, Lemma F.1] with σ 2 = 1 / 4 as we here consider [0 , 1] -bounded rewards. Note that the concentration bounds in [Dann et al., 2017] apply for the conditionally subgaussian random variables, and one can get the same result for i.i.d. subgaussian random variables, by simply replacing the Doob's maximal inequality by Hoeffding's maximal inequality (see [Jamieson et al., 2014, Lemma 3]).

One caveat here is that our analysis still works for other choices of U δ ( x ) if one adjust constant or change log log( · ) to log( · ) .

Algorithm 9 lil'UCB

E.1 Proof of Theorem 3.3

In the following proof, we consider the following instance.

Definition E.1 (Two-armed bandit instance) . Consider a two-armed setting with µ 1 > µ 2 . In each round, each arm generates deterministic rewards µ 1 and µ 2 , respectively. The arm gap is ∆ = µ 1 -µ 2 and ∆ ∈ (0 , 0 . 6) .

According to Lemma E.2, the total number of plays of arm 2 is finite. Therefore, one can find a round t 0 ∈ N that the last play of arm 2 occurs. At the beginning of this round, the algorithm compares µ 1 + U δ ( N 1 ( t 0 )) and µ 2 + U δ ( N 2 ( t 0 )) . Since arm 2 gets the last play at this round, we have

where the last inequality holds due to Lemma E.4 and f (∆) is defined as

Rearranging the above, we have

which immediately leads to

Moreover, since arm 2 is played at round t 0 , ∆ = ∆ A t 0 , which further implies that

̸

where the third inequality uses f (∆) ≤ 1 for all ∆ ∈ (0 , 0 . 6) , and the last equality holds due to ∆ = ∆ A t 0 (recall that in round t 0 , arm 2 gets the last play).

Rearranging the above, we have

Finally, by Lemma E.3, we have t 0 = Ω(∆ -2 ) , which completes the proof.

E.2 Proof of Supporting Lemmas

Lemma E.2. Suppose that we run Algorithm 9 for the two-armed bandit instance given in Definition E.1. For any δ ∈ (0 , 1) , suboptimal arm 2 will be played at most

Proof. Note that for any fixed δ ∈ (0 , 1) , function U δ ( x ) is monotonically-decreasing for x ≥ 6 . To show the claimed result, it suffices to find an integer x ≥ 6 such that

Now, we consider the following choice.

Notice that N ≥ n ≥ 6 for all ∆ ∈ (0 , 1] . Then, one can show

where the first inequality uses the fact that for any fixed δ ∈ (0 , 1) , U δ ( x ) is monotonically-decreasing for x ≥ 6 , the second inequality uses x + y ≤ 2 xy for all x, y > 1 (here x = log(6 /δ ) and y = log log ( 16 log(6 /δ ) ∆ 2 ) ), the last inequality first bounds 1 4 log log ( 16 log(6 /δ ) ∆ 2 ) ≤ 16 log(6 /δ ) ∆ 2 and then uses 2 x ≤ x 2 for all x ≥ 2 with x = log ( 16 log(6 /δ ) ∆ 2 ) ≥ 2 .

Thus, the proof is complete.

From Lemma E.2, the total number of plays of suboptimal arm is finite. Based on this fact, we present the following lemmas.

Lemma E.3. Suppose that we run Algorithm 9 for the two-armed bandit instance given in Definition E.1. Let n +1 be the total number of plays of arm 2 . Then,

Proof. For shorthand, we define

With this definition and ∆ ∈ (0 , 0 . 6) , we have m ≥ 6 and ⌊ m ⌋ ≥ 6 .

Now, we show that n ≥ ⌊ m ⌋ . To this end, it suffices to show U δ ( ⌊ m ⌋ ) > ∆ . As U δ ( x ) is monotonically-decreasing for all x ≥ 6 and m ≥ ⌊ m ⌋ ≥ 6 , we have

where the last inequality holds as log(6 /δ ) ∆ 2 > log(6) ∆ 2 > e .

Once U δ ( ⌊ m ⌋ ) > ∆ holds, ⌊ m ⌋ cannot be the last play of arm 2 (i.e., at least the one before the last), which completes the proof.

Lemma E.4. Suppose that we run Algorithm 9 for the two-armed bandit instance given in Definition E.1. Let n +1 be the total number of plays of arm 2 . The following holds.

Proof. From Lemma E.3, we have n ≥ 6 , and thus U δ ( n ) ≥ U δ ( n +1) as U δ ( x ) is monotonically decreasing for all x ≥ 6 . Using the fact that for all a ≥ b ≥ 0 , a -b ≤ √ a 2 -b 2 (here a = U δ ( n ) and b = U δ ( n +1) ), one can show that

Since for any fixed δ ∈ (0 , 1) , U δ ( x ) is monotonically-decreasing for all x ≥ 6 , and thus by using the definition of m (see Eq. (18)), one can show that

where the first inequality uses Eq. (19) with n ≥ ⌊ m ⌋ ≥ e and U δ ( x ) is monotonically decreasing for all x ≥ 6 , the second inequality upper-bounds ⌊ m ⌋ ≤ m in numerator and lower-bounds ⌊ m ⌋ ( ⌊ m ⌋ +1) ≥ m 2 , the third inequality follows from

and the fourth inequality follows 2 x ≤ x 3 for all x ≥ √ 2 with x = log ( log(6 /δ ) ∆ 2 ) ≥ √ 2 ( ∆ ∈ (0 , 0 . 6) ).

After arm 2 gets n +1 times play, it will not be played anymore, which implies that U δ ( n +1) < ∆ . Using this bound gives

Therefore, we complete the proof.

E.3 Traditional Optimistic Algorithms Fail to Achieve ULI Guarantee

We will show that the traditional optimistic algorithms fail to achieve the ULI guarantee due to the log t term in the bonus. At a high level, log t is increasing with time and forces the algorithm to play a bad arm indefinitely when t evolves. We provide the formal proof in the following:

Weprove the claim by contradiction. Consider a K -armed bandit instance with at least one suboptimal arm, and let ∆ > 0 be the minimum arm gap. Suppose there exists an optimism-based algorithm with log t in bonus term that can achieve the ULI guarantee in this setting. Then, for some fixed δ ∈ (0 , 1) , with probability ≥ 1 -δ , for all t ∈ N , ∆ t ≤ F ULI ( δ, t ) . Based on Definition 2.5, we have lim t →∞ F ULI ( δ, t ) = 0 and F ULI ( δ, t ) is monotonically decreasing w.r.t. t after a threshold. Thus, ∃ t 0 ∈ N such that F ULI ( δ, t ) < ∆ for all t ≥ t 0 . In other words, the algorithm cannot play any suboptimal arm after t 0 -th round. Recall that the bonus term is √ log t/N a ( t ) where N a ( t ) is the number of plays of arm a before round t . For any suboptimal arm a , N a ( t ) should not increase after t 0 -th round, but log t keeps increasing. This leads the bonus of arm a goes to infinity, which will incur a play of arm a at a round after t 0 . This makes a contradiction.

Algorithm 10 Meta-algorithm towards ULI

Input : δ ∈ (0 , 1) , finite arm set A , and base-algorithm Alg with function g ( δ ) that satisfies Condition. F.1.

:

Initialize

A

1

= [

K

1 for m = 1 , 2 , . . . do

2

Set duration

T

m

]

.

=

⌈

576

g

m

2

⌉

where

g

m

=

g

δ

2

m

2

.

3 Run Alg with active arm set A m for T m rounds and construct ̂ ℓ t,a = ℓ t,a · I { A t = a } p t,a for all t, a in this phase.

4 Update active arm set A m +1 = A m -B m where:

and T m is a set that contains all rounds in phase m and k is the empirical best arm:

F Achieving ULI by Adversarial Bandit Algorithms

F.1 Meta-algorithm Enabling Adversarial Algorithms to Achieve ULI

In this subsection, we propose a meta-algorithm shown in Algorithm 10 which enables any highprobability adversarial algorithm, with a mild condition, to achieve the ULI guarantee. Then, we show that existing high-probability adversarial bandit algorithms naturally meet this condition in both MAB and linear bandit settings. For notational simplicity, we follow the convention of adversarial analysis to use loss ℓ t,a = 1 -X t,a . Note that all algorithms in this subsection require [0 , 1] -boundedness assumption on loss.

At a high-level, our meta-algorithm keeps running a base-algorithm Alg to play arms and collects rewards. The meta-algorithm uses collected rewards to construct the importance-weighted (IW) estimator ̂ ℓ t,a = ℓ t,a · I { A t = a } p t,a for each t, a , which helps to eliminate bad arms, and then runs Alg on a reduced arm space. To achieve the ULI guarantee, our meta-algorithm requires the input base-algorithm to satisfy the following:

Condition F.1. An algorithm Alg runs for given consecutive T rounds with a finite arm set A and a fixed δ ∈ (0 , 1) . At each round t ∈ [ T ] , Alg maintains a distribution p t over A and samples an arm A t ∼ p t . With probability at least 1 -3 5 δ , Alg ensures for all a ∈ A ,

where g ( δ ) is a positive-valued function, monotonically decreasing for all δ ∈ (0 , 1) , polynomial in log(1 /δ ) , and g ( δ ) ≥ log (10 |A| /δ ) , ∀ δ ∈ (0 , 1) .

In fact, many high-probability adversarial bandit algorithms naturally meet Condition. F.1, but require stronger analysis. In particular, we show that EXP3.P [Auer et al., 2002b] for MAB holds this condition with g ( δ ) = O ( |A| log( |A| /δ ) ) (omitted proof can be found in Appendix F) and Lee et al. [2021] show that GEOMETRICHEDGE.P [Bartlett et al., 2008] with John's exploration meets the condition for linear bandits with g ( δ ) = O ( d log( |A| /δ ) ) . Hence, feeding those algorithms to Algorithm 10 yields the following:

Theorem F.2. For any fixed δ ∈ (0 , 1) , if Algorithm 10 uses

EXP3.P as a base-algorithm, then, for K -armed bandit, ULI guarantee is achieved with

m

(

)

GEOMETRICHEDGE.P as a base-algorithm then for linear bandits with K arms, ULI guarantee is achieved with

Theorem F.2 suggests that Algorithm 10 enables EXP3.P and GEOMETRICHEDGE.P to achieve nearoptimal ULI guarantees for MAB and linear bandits, respectively. Moreover, their ULI guarantees are as good as those of conventional elimination-based algorithms (refer to Theorem 3.2).

Our main objective in this section is to prove the following results.

Theorem F.3 (Restatement of Theorem F.2) . For any fixed δ ∈ (0 , 1) , if Algorithm 10 uses

GEOMETRICHEDGE.P as a base-algorithm then for linear bandits with K arms, ULI guarantee is achieved with

We decompose the proof of Theorem F.2 into two parts. In the first part, we first show that any algorithm with Condition. F.1 for some δ ∈ (0 , 1) can achieve the ULI guarantee. Then, we only need to show that EXP3.P and GEOMETRICHEDGE.P satisfy Condition. F.1, which completes the proof of Theorem F.2.

The following result suggests that showing an algorithm enjoys the ULI guarantee is reduced to show that this algorithm meets Condition. F.1.

Theorem F.4. For any fixed δ ∈ (0 , 1) , if Algorithm 10 uses a base-algorithm that satisfies Condition. F .1, then, it achieves the ULI guarantee that

We sketch the proof of Theorem F.4 as follows. The full proof can be found in Appendix F.2.

Proof Sketch. The high-level objectives are to show the sufficient condition of Eq. (1) given in Algorithm 1 to achieve the ULI. We first show that the optimal arm will not be eliminated, i.e., a ⋆ ∈ A m for all phases m ∈ N . To show this, we only need show

which does not meet the elimination rule in Eq. (20).

For analysis purpose, we decompose Eq. (22) as

The first and the third term in Eq. (23) can be handled thanks to the bound on ∣ ∣ ∣ ∑ t ( ̂ ℓ t,a -ℓ t,a ) ∣ ∣ ∣ in Eq. (21), and the second term in Eq. (23) can be handled by invoking the standard concentration inequality as losses are drawn from fixed distributions.

Then, we show that if an arm is still active at a round t , then, for all active arms a ∈ A t , ∆ a is bounded by a function, monotonically decreasing for large t . This can be proved again via a similar decomposition of Eq. (23) to get

Therefore, for some large T m such that 0 . 5 T m ∆ a -5 √ g m T m > 7 √ g m T m , bad arms will be eliminated. Putting two pieces together, we complete the proof.

With Theorem F.4 in hand, our next objective is to show that EXP3.P and GEOMETRICHEDGE.P are able to satisfy Condition. F.1. For GEOMETRICHEDGE.P with John's exploration (which improves the original bound [Bartlett et al., 2008]), Lee et al. [2021] have already shown that it achieves the condition, and the rest result will show that EXP3.P also holds it. The full proof can be found in Appendix F.3.

Proposition F.5. In MAB setting, for any δ ∈ (0 , 1) , EXP3.P with given arm set A , satisfies Condition. F.1 with g ( δ ) = O ( |A| log ( |A| /δ )) .

The key that EXP3.P as well as GEOMETRICHEDGE.P can fulfill Condition. F.1 is because it adds a small amount of probability for uniform exploration to each arm. Therefore, the importance-weighted (IW) estimator can be lower-bounded and the term ∣ ∣ ∣ ∑ t ( ̂ ℓ t,a -ℓ t,a ) ∣ ∣ ∣ will not be too large.

Apart from EXP3.P for MAB and GEOMETRICHEDGE.P for linear bandits, Lee et al. [2021] show that refined version of SCRIBLE [Lee et al., 2020] can be used as a base-algorithm for Algorithm 10 to achieve the ULI guarantee for linear bandits, but the ULI guarantee is inferior to that of GEOMETRICHEDGE.P (i.e., inferior F ULI w.r.t. d ).

F.2 Proof of Theorem F.4

Lemma F.6. If Algorithm 10 accepts a base-algorithm which satisfies Condition. F.1 as an input, then, with probability at least 1 -3 5 δ , for all m ∈ N and a ∈ A m ,

Proof. We first consider a fixed phase m . Since the base-algorithm satisfies Condition. F.1, if the base-algorithm runs for consecutive T m and active arm set A m ⊆ A , then, with probability at least 1 -3 5 δ ′ , for all a ∈ A m

By setting δ ′ = δ/ (2 m 2 ) for phase m and applying a union bound over all m ∈ N , we complete the proof.

Recall from the second bullet of Condition. F.1 that

As the above holds for all δ ′ ∈ (0 , 1) , according to the definition g m = g ( δ/ (2 m 2 )) , we also have that

Lemma F.7. With probability at least 1 -δ/ 5 , for all m ∈ N and a ∈ A m , we have

Proof. Consider any fixed m and a ∈ A m . By applying Hoeffding's inequality with probability at least 1 -δ ′ ,

Choosing δ ′ = δ/ (10 |A| m 2 ) , applying union bounds over all m ∈ N and a ∈ A m , and using Eq. (25), we complete the proof.

Lemma F.8. With probability at least 1 -δ/ 5 , for all m ∈ N , we have

Proof. Consider a fixed m . For each round s in this phase, let us define

By applying Azuma-inequality with the fact that | M s -M s -1 | ≤ 2 for all s and also using a union bound over all m ∈ N , with probability at least 1 -δ/ 5 , for all m ∈ N

where the last inequality uses Eq. (25), which thus completes the proof.

Definition F.9. Let E be the event in which the concentration inequalities in Lemma F .6, Lemma F .7, and Lemma F.8 hold simultaneously.

By a union bound, we have the following fact.

Fact. With the definition of E in Definition F.9, E occurs with probability at least 1 -δ .

Lemma F.10. Suppose that E occurs where E is defined Definition F .9, and then for all m ∈ N and a ∈ A m ,

Proof. Conditioning on E , for each m ∈ N and a ∈ A m , one can show

where the first inequality holds due to Lemma F.6, the second inequality uses Lemma F.7 and Lemma F.8, and the last inequality bounds µ j ≤ µ ⋆ for all j .

Corollary F.11. Suppose that E occurs where E is defined Definition F .9, and then for all m ∈ N and a ∈ A , we have

Proof. The proof is immediate by applying Lemma F.7.

Lemma F.12. Suppose that E occurs where E is defined Definition F .9, and then a ⋆ ∈ A m for all m ∈ N .

Proof. We prove the claimed result by induction. For the base case, the claim holds for the first phase m = 1 as A 1 = A . Suppose that a ⋆ ∈ A m . Then, we show that a ⋆ will not be eliminated at the end of phase m , thereby active for phase m +1 . Then, we have

where the first inequality uses Lemma F.10 together with ∆ a ⋆ = 0 and Corollary F.11.

Once the induction is done, we complete the proof (recall the elimination rule in Algorithm 10).

Lemma F.13. Suppose that E occurs where E is defined Definition F .9. For each arm a ∈ A with ∆ a > 0 , it will not be in A m for all m ≥ m a + 1 , where m a is the smallest phase such that 2 m a > 1 ∆ 2 a .

Proof. Consider fixed phase m and arm a ∈ A with ∆ a > 0 . Suppose that arm a is still active in an phase m . One can show

where the first inequality holds since Lemma F.12 implies that a ⋆ ∈ A m for all m ∈ N , and the second inequality uses Lemma F.10 together with ∆ a ⋆ = 0 and Corollary F.11.

Let m a be the minimum phase such that 2 m a > 1 ∆ 2 a (i.e., 2 m a -1 ≤ 1 ∆ 2 a ), which further gives that

Hence, by the definition of m a , we have T m > 576 g m ∆ 2 a for all m ≥ m a , which gives that T m ∆ a > 24 √ g m T m for all m ≥ m a . Plugging this into Eq. (30), we arrive at

which implies that arm a will not be active in all phases m ≥ m a +1 according to the elimination rule.

Finally, one can repeat this argument for each a ∈ A with ∆ a > 0 conditioning on E .

Lemma F.14. Let m ( t ) be the phase in which round t lies. Then, m ( t ) ≤ log 2 ( t +1) for all t ∈ N .

Proof. We prove this by contradiction. Suppose that ∃ t ∈ N that m ( t ) > log 2 ( t +1) . Note that we can further assume m ( t ) ≥ 2 since one can easily verify that for all t such that m ( t ) = 1 ,

m ( t ) ≤ log 2 ( t +1) must hold. Recall that in phase m ( t ) , each active arm will be played for m ℓ ( t ) times, we have

where the third inequality bounds ℓ ( t ) > log 2 ( t +1) by assumption. Therefore, once a contradiction occurs, the proof is complete.

Lemma F.15. Let m ( t ) be the phase in which round t lies. Suppose that E occurs. For all t ∈ N and all a ∈ A , if a ∈ A m ( t ) , then,

Proof. If a ∈ A m ( t ) is optimal, then, ∆ a = 0 and the claim trivially holds. In what follows, we only consider arm a ∈ A m ( t ) with ∆ a > 0 . Then, t can be bounded by

where the second inequality simply bounds ⌈ 576 g n 2 n ⌉ ≤ 2 × 576 g n 2 n for all phases n , the fourth inequality holds because Lemma F.13 implies that if a ∈ A m ( t ) , then, m ( t ) ≤ m a holds, and the last inequality uses 1 ∆ 2 a ≥ 2 m a -1 .

Since for any fixed δ ∈ (0 , 1) , g ( δ/x 2 ) is monotonically increasing for x ≥ 1 (recall Condition. F.1) and Lemma F.14 gives m ( t ) ≤ log 2 ( t +1) , we have

which gives t ≤ 4608 g ( δ/ (2 log 2 2 ( t +1))) ∆ 2 a . Conditioning on E , this argument holds for each t, a , which completes the proof.

Proof of Theorem F.4. Once Lemma F.12 and Lemma F.15 hold, and g ( x ) is polynomial in log(1 /x ) , Theorem 3.1 gives that for any fixed δ ∈ (0 , 1) , Algorithm 10 achieves the ULI guarantee with a function

According to the first bullet of Condition. F.1, the proof is complete.

F.3 Proof of Proposition F.5

In this section, we prove that EXP3.P [Auer et al., 2002b] meets Condition. F.1. In our setting, arm set is A = [ K ] , and the loss ℓ t,a generated by each arm a at each round t is assumed to be [0 , 1] -bounded.

Algorithm 11 EXP3.P

Input : Time horizon T , arm set [ K ] , confidence δ ∈ (0 , 1) . Initialize : ∀ a ∈ [ K ] , w 1 ,a = 1 and parameters η = η δ ( T ) , γ = γ δ ( T ) , β = β δ ( T ) according to Eq. (33).

for

t

= 1

,

2

, . . . , T

do

Play an arm A t ∈ [ K ] from distribution p t = [ p t, 1 , . . . , p t,K ] and observe loss ℓ t,A t where

Update w t +1 ,a = w t,a exp( -η ˜ ℓ t,a ) for all a ∈ [ K ] with

The pseudocode of EXP3.P is given in Algorithm 11 and we briefly review its procedure. Ahead of time, EXP3.P accepts a fixed time horizon T ∈ N arm set [ K ] , and confidence δ ∈ (0 , 1) as inputs. At each round t ∈ [ T ] , EXP3.P pulls an arm A t ∼ p t from a distribution p t = [ p t, 1 , . . . , p t,K ] , and then observes the loss ℓ t,A t . The probability of playing an arm a at round t is (1 -γ ) w t,a W t + γ K where γ > 0 is a fixed parameter, which encourages the exploration, w t,a is the weight of arm a , and W t = ∑ a ∈ [ K ] w t,a . After pulling the arm, the algorithm uses the observed reward to construct the shifted IW-estimators ˜ ℓ t,a according to Eq. (32), and finally uses the shifted IW-estimators to update the weight w t,a for each arm.

The parameters of γ δ ( T ) , η δ ( T ) , β δ ( T ) are as a function T , given as

Let E t [ · ] be the conditional expectation given the history prior to round t .

Lemma F.16 (Exercise 5.15 of [Lattimore and Szepesvári, 2020]) . Let { X t } T t =1 be a sequence of random variables adapted to filtration {F t } T t =1 and let β > 0 such that βX t ≤ 1 almost surely for all t ∈ [ T ] . With probability at least 1 -δ ,

Corollary F.17. With probability at least 1 -δ/ 5 , for all a ∈ [ K ] ,

Proof. Consider a fixed arm a . For all t, a , we have ℓ t,a ∈ [0 , 1] , and thus

Then, we check β ̂ ℓ t,a ≤ 1 almost surely for all t ∈ [ T ] . Specifically, we use p t,a ≥ γ/K to show

where the last equality holds due to β = γ/K according to Eq. (33).

For the fixed a , applying Lemma F.16 with X t = ̂ ℓ t,a , we have that with probability at least 1 -δ ′

Choosing δ ′ = δ/ (5 K ) and applying a union bound over all a ∈ [ K ] yield the claimed bound.

Lemma F.18. With probability at least 1 -δ/ 5 , for all a ∈ [ K ] , we have

Proof. Consider any fixed arm a . By applying Hoeffding's inequality with probability at least 1 -δ ′ ,

Choosing δ ′ = δ/ (5 K ) and applying a union bound over all arms, we complete the proof.

Lemma F.19 (Freedman's inequality) . Let { X t } T t =1 be a martingale difference sequence with respect to filtration {F t } T t =1 and | X t | ≤ M almost surely for all t . Then, for any δ ∈ (0 , 1) , with probability at least 1 -δ ,

Corollary F.20. With probability at least 1 -δ/ 5 , the following holds for all a ,

Proof. Consider a fixed arm a . Let M t,a = ̂ ℓ t,a -(1 -µ a ) and { M t,a } T t =1 is a martingale difference sequence. We have E t [ M t,a ] = 0 , | M t,a | ≤ K γ and also

where the last step holds due to Eq. (34).

By Lemma F.19 and a union bound over all a , with probability at least 1 -δ ′ for all a

Choosing δ ′ = δ/ 5 completes the proof.

Definition F.21. Let E 0 be the event in which all inequalities of Corollary F .17, Lemma F .18, and Corollary F.20 hold simultaneously. With this definition, E 0 occurs with probability at least 1 -3 δ 5 . Lemma F.22. Let θ 0 be an arbitrary constant such that θ 0 ≥ 1 . Suppose that E 0 occurs where E 0 is defined in Definition F .21, and then for all a ∈ [ K ] , we have

Proof. Let θ 0 > 0 be an arbitrary constant. By Corollary F.20, for all a ∈ [ K ]

√

where the second inequality uses 2 ab ≤ a + b for all a, b ≥ 0 . Moreover, by the triangle inequality, we have

where the last inequality applies Lemma F.18 and uses Eq. (38). Rearranging the above gives

Finally, if we constrain θ 0 ≥ 1 , then

where the first inequality uses Eq. (39) and the last one bounds θ 0 ≥ 1 . Since this argument holds for all a ∈ [ K ] conditioning on E 0 , we complete the proof.

Lemma F.23. The following holds for all t ∈ [ T ] .

Proof. For any fixed t , we can show

where the first inequality uses ̂ ℓ t,a -4 β p t,a ≤ ̂ ℓ t,a -2 β p t,a = ˜ ℓ t,a and p t,a ≥ γ K , and the last inequality bounds p t,a ̂ ℓ t,a ≤ 1 for each a ∈ [ K ] . Since the claim deterministically holds for all t , the proof is complete.

Lemma F.24. The following holds for all arms k ∈ [ K ] .

Proof. Let us define

With this definition, one can show for t ≥ 2

where the first inequality uses e -x ≤ 1 -x + x 2 whenever x ≥ -1 (here we repeat x = η ˜ ℓ t,a for every a ), and the second inequality follows from log(1 + x ) ≤ x for all x > -1 .

By summing over all t and using Lemma F.23 with η = γ K , the above result yields

As Φ 0 = 0 and 1 -γ ≤ 1 , we have for an arbitrary arm k ∈ [ K ] ,

As this argument deterministically holds for all k ∈ [ K ]

, we complete the proof.

Proof of Proposition F.5. The following analysis will condition on event E 0 which is defined in Definition F.21. As mentioned in Definition F.21, this event occurs with probability at least 1 -3 5 δ . We first note that for all t, a

Recall that we choose parameters as

In what follows, we consider T > 4 K log(10 K/δ ) 8 . In this case, we have η, γ, β ∈ (0 , 1 2 ] . Then, for an arbitrary arm k ∈ [ K ] , we pick θ 0 = 2 and show

where the equality uses Eq. (40) and the inequality applies Lemma F.22.

We first apply Lemma F.24 to bound TERM 1 as

where the second inequality uses Corollary F.17, the third inequality uses ∑ i ℓ t,a ≤ K , and the last inequality uses T ≥ 4 K log(10 K/δ ) (i.e., K ≤ √ KT 4 log(10 K/δ ) ).

Then, we use K/γ = 1 /η to bound TERM 2:

Finally, we apply Eq. (36) with θ 0 = 2 to bound TERM 3.

Putting bounds of TERM 1, TERM 2, and TERM 3 together gives

As this argument holds for all k ∈ [ K ] conditioning on E 0 , the proof is thus complete.

F.4 Proof of Theorem F.2

For EXP3.P, we apply g ( δ ) = O ( K log ( K/δ )) given in Proposition F.5 into Theorem F.4 to get the claimed result.

As for GEOMETRICHEDGE.P, we consider an improved version in [Lee et al., 2021] which uses John's exploration and shows that g ( δ ) = d log( K/δ ) . Note that this is slightly different from d log( K log 2 T/δ ) presented in [Lee et al., 2021]. The extra log 2 T term is caused by Lemma 2 of [Bartlett et al., 2008], a concentration inequality for martingales, but one can invoke Lemma F.19 to avoid it. By applying such a function g ( δ ) into Theorem F.4, we obtain the claimed result.

Algorithm 12 Adaptive barycentric spanner

Input : compact arm set A , phase m , spanner B m -1 , parameters T m , C , and estimation ̂ θ m . Initialize : matrix A = [ a 1 , . . . , a d ] by setting a i = e i √ ∈ R d for all i ∈ [ d ] , and I m = [ d ] .

̸

̸

G Omitted Details of Section 4

G.1 Notations

We use e i ∈ R d to denote a vector whose i -th coordinate is one and all others are zero. For any two vectors x, y ∈ R d , x ≤ y indicates that x i ≤ y i holds for each coordinate i . For a positive definite matrix A ∈ R d × d , the weighted 2 -norm of vector x ∈ R d is given by ∥ x ∥ A = √ x ⊤ Ax . For matrix A = [ a 1 , · · · , a d ] ∈ R d × d with each a i ∈ R d , we use A -i to denote the ( d -1) -tuple of vectors [ a 1 , · · · , a i -1 , a i +1 , · , a d ] . For matrices A,B , we use A ≻ B to indicate that A -B is positive definite. For two sets A , B , we use A-B to indicate the exclusion. For any scalar C ∈ R , we use ⃗ C to denote a vector, with all coordinates equal to C .

G.2 Key Technique: Adaptive Barycentric Spanner

Algorithm description. Algorithm 12 aims to identify a finite arm set B m to linearly represent the (possibly infinite) active arm set A m , so that playing arms in B m allows us to obtain an accurate estimation of ⟨ a, θ ⟩ for each a ∈ A m .

Algorithm 12 initializes a matrix A = [ a 1 , . . . , a d ] whose column vectors are linearly independent, and a i = e i / √ T m . At the beginning, the algorithm solves (42) to find an empirical best arm a ⋆ m (line 1-6). Then, in two for-loops (line 7-10 and line 11-16), the algorithm tries to replace each column vector of A with an active arm a ∈ A m by invoking the subroutine LI-Argmax in Algorithm 13, while keeping column vectors of A linearly independent. For each column i , LI-Argmax attempts to find max a ∈A m | det( a, A -i ) | 9 by solving (43) with a ⋆ m . If max a ∈A m | det( a, A -i ) | = 0 , then,

Algorithm 13 Linear-independent argmax (LI-Argmax)

Input : compact arm set A , matrix A , estimation ̂ θ m , spanner B m -1 , and phase m , parameter C , and arm a ⋆ m .

1 Find u ∈ R d s.t. ⟨ u, x ⟩ = det ( x, A -i ) , ∀ x ∈ R d .

2 Query oracle to obtain a + , the solution of

3 Query oracle to get a from (43) by replacing u with -u .

the algorithm returns NULL, since there does not exist an active arm that can maintain the linear independence. In this case, the algorithm keeps e i / √ T m in the i -th column of A . Otherwise, a i will be updated to s i ∈ argmax a ∈A m | det( a, A -i ) | . We use set I m to record the indices of columns where the replacement fails. According to the conditions in line 14, the second for-loop will be repeatedly restarted to ensure that the linear combination of elements in B m can represent all active arms in A m with coefficients in the range of [ -C, C ] . Also note that, as shown in Theorem 4.2, LI-Argmax will be oracle-efficient as it only queries the oracle ˜ O ( d 3 ) times.

Formally, we have the following result.

Lemma G.1. Suppose C > 1 and B m is the output of Algorithm 12 in phase m . Then, for all m ∈ N , B m is a C -approximate barycentric spanner of A m .

G.3 Computational Analysis

Recall from the previous subsection that Algorithm 12 and LI-Argmax needs to find the empirical best arm a ⋆ m and argmax a ∈A m | det( a, A -i ) | , both of which rely on the access to the following optimization oracle.

Definition G.2 (Optimization oracle) . Given a compact set A ⊆ R d , the oracle can solve problems of the form

for any β 1 , β 2 ∈ R , θ ∈ R d , U ∈ R τ 1 × d , V ∈ R τ 2 × d , where τ 1 , τ 2 are at most O ( d ) . If the optimal solution is not unique, the oracle returns any one of them.

We note that the constrained optimization oracles are commonly-used and also crucial for eliminationtype approaches e.g., [Bibaut et al., 2020, Li et al., 2022]. Although this oracle is slightly powerful than that used for non-elimination based approaches [Dani et al., 2008, Agarwal et al., 2014], linear constrained oracle, in fact, can be implemented efficiently in many cases (e.g., a common assumption that A is a ball [Plevrakis and Hazan, 2020]) via the ellipsoid method. We refer readers to [Bibaut et al., 2020, Plevrakis and Hazan, 2020] for more discussions.

With the optimization oracle in hand, our goal here is to query the oracle to solve the optimization problem argmax a ∈A m | det( a, A -i ) | , which can be rewritten as follows based on Eq. (3).

Problem (44) also needs to find the empirical best arm a ⋆ m . A natural idea to compute the empirical best arm a ⋆ m is to solve argmax a ∈A m -1 ⟨ ̂ θ m , a ⟩ . Notice that both finding a ⋆ m and solving Eq. (44)

require to deal with constraint a ∈ A m -1 . However, this constrain prevents us from querying the optimization oracle due to the following reason.

Issue: a ∈ A m -1 may cause a large (even infinite) number of constraints. One can rewrite a ∈ A m -1 as:

However, the evolution of phases blows up the number of constraints in (45) (e.g., when m = Ω(exp( d )) ), which prevents LI-Argmax from directly querying the optimization oracle defined in Definition G.2, since it requires the number of constraints at most O ( d ) . To address this issue, one needs to remove the dependence on m .

Solution: enlarging active arm set A m -1 . Our solution is to slightly enlarge the active arm set A m -1 in Eq. (45) so that the enlarged set can be expressed by finitely-many linear constraints , independent of m . Before showing the way to enlarging A m -1 , we first give the following definition.

Definition G.3 ( C -bounded spanner) . For any given set S ⊆ R d and constant C > 0 , Span [ -C,C ] ( S ) , defined as follows, is a set that contains all possible linear combinations from S with coefficients within [ -C, C ] .

where c s is the coefficient associated with element s .

We enlarge A m -1 to A ∩ Span [ -C,C ] ( B m -1 ) , and one can easily verify that this is true since A m -1 ∩ A = A m -1 and Lemma G.1 implies that A m -1 ⊆ Span [ -C,C ] ( B m -1 ) .

Now, it remains to show that Span [ -C,C ] ( B m -1 ) can be expressed by O ( d ) number of linear constraints. The following lemma gives the desired result.

Lemma G.4. Suppose that |B m -1 | < d . For M m and B m -1 given in Algorithm 12, Span [ -C,C ] ( B m -1 ) can be equivalently written as

Hence, LI-Argmax can invoke the optimization oracle in Definition G.2 to solve problems of argmax a ∈A m -1 ⟨ ̂ θ m , a ⟩ and (44) approximately by solving (42) and (43), respectively.

Computational efficiency. We mainly focus on two costly steps in Algorithm 12. The first step is to use Gaussian elimination to obtain the matrix M m ∈ R n × d for some n ≤ d . This step can be done with at most O ( d 3 ) complexity and is implemented at most once for each phase m . The other expensive step is to query the optimization oracle to construct a C -approximate barycentric spanner. The number of calls to the oracle depends on the number of times that Algorithm 12 invokes LI-Argmax subroutine. The LI-Argmax will be invoked for d times in the first for-loop. For the second for-loop, LI-Argmax will be invoked for ˜ O ( d 3 ) times. Compared with ˜ O ( d 2 ) in [Awerbuch and Kleinberg, 2008], the extra d in our complexity comes from the additional restart-condition i ∈ I m in line 14 of Algorithm 12. As |I m | ≤ d and I m is non-increasing when update, this condition will be met for at most d times.

G.4 Proof of Lemma G.4

Lemma G.5 (Restatement of Lemma G.4) . Suppose that |B m -1 | < d . For M m and B m -1 given in Algorithm 12, the set Span [ -C,C ] ( B m -1 ) = H m where

Proof. For this proof, we show that a ∈ Span [ -C,C ] ( B m -1 ) if and only if a ∈ H m .

We first show that if a ∈ Span [ -C,C ] ( B m -1 ) , then a ∈ H m . As matrix M m ∈ R ( d -|B m -1 | ) × d is defined by Span ( B m -1 ) = { x ∈ R d : M m x = ⃗ 0 } , a ∈ Span [ -C,C ] ( B m -1 ) ⊆ Span ( B m -1 ) gives

that M m a = ⃗ 0 . Recall that B m -1 ∈ R d ×|B m -1 | is a matrix whose the i -th column is the i -th element of set B m -1 and rank ( B m -1 ) = |B m -1 | because B m -1 is linearly independent. Therefore, for each a ∈ Span [ -C,C ] ( B m -1 ) , there exists a unique vector 10 x a ∈ R |B m -1 | such that B m -1 x a = a and the value of each coordinate of x a is no larger than C . As B m -1 is linearly independent, B m -1 is a full-column rank matrix, which gives that x a = ( B ⊤ m -1 B m -1 ) -1 B ⊤ m -1 a . As a result, if a ∈ Span [ -C,C ] ( B m -1 ) , then, -⃗ C ≤ ( B ⊤ m -1 B m -1 ) -1 B ⊤ m -1 a ≤ ⃗ C , which concludes the proof of this argument.

Then, we show that if a ∈ H m , then, a ∈ Span [ -C,C ] ( B m -1 ) . If M m a = ⃗ 0 , then, we have a ∈ Span ( B m -1 ) due to the definition of M m . As rank ( B m -1 ) = |B m -1 | , for each a ∈ Span ( B m -1 ) , there exists a unique vector x a ∈ R |B m -1 | such that B m -1 x a = a , which gives that x a = ( B ⊤ m -1 B m -1 ) -1 B ⊤ m -1 a . Thus, -⃗ C ≤ ( B ⊤ m -1 B m -1 ) -1 B ⊤ m -1 a ≤ ⃗ C requires each coordinate of x a no larger than C . As a consequence, if a ∈ H m , then, a ∈ Span [ -C,C ] ( B m -1 ) .

Combining the above analysis, we complete the proof.

G.5 Proof of Lemma G.1

In this section, we aim to show that B m is a C -approximate barycentric spanner of A m . To this end, we first show that B m is a C -approximate barycentric spanner of S m which is defined below and A m ⊆ S m results in the desired claim. In fact, throughout our analysis, we stick with S m rather than A m .

Moreover, according to Lemma G.4, for all m ≥ 2 , S m can also be equivalently rewritten as:

Lemma G.6. Let C > 1 and B m is in Algorithm 2. For all m ∈ N , B m is a C -approximate barycentric spanner of S m .

Proof. For all m ∈ N , we define B ′ m = B m ∪ ( ∪ i ∈I m e i √ T m ) . We further define set S ′ m = S m ∪ ( ∪ i ∈I m e i √ T m ) , and matrix B ′ m = [ b 1 , · · · , b d ] where b i is the i -th element of B ′ m . By these definitions, b i = e i √ T m for all i ∈ I m . The proof consists of three steps, and we first proceed step 1.

Step 1: each solution of (43) is in S m . According to the optimization problem (43), for each phase m , the solution is drawn from (see Eq. (47) for definition of H m )

where the first equality holds due to Lemma G.4 and the last equality holds due to Eq. (48).

Step 2: B ′ m is a C -approximate barycentric spanner of S ′ m . The proof idea of this step follows a similar arguments of Proposition 2.2 and Proposition 2.5 in [Awerbuch and Kleinberg, 2008]. In Algorithm 12, each update for matrix A ∈ R d × d always maintains | det( A ) | > 0 , and thus B ′ m spans R d . As S ′ m ⊆ R d , every element a ∈ S ′ m can be linearly represented by a = ∑ d i =1 w i b i for some coefficients { w i } d i =1 , and then we have

Step 1 shows that each solution of (43) is in S m , and thus B m ⊆ S m , which further implies B ′ m ⊆ S ′ m . Recall that before the termination, the algorithm will scan each column vector of matrix A to ensure that no more replacement can be made. Hence, once Algorithm 12 terminates, we have that ∀ i ∈ [ d ] , sup a ∈S ′ m | det( a, ( B ′ m ) -i ) | ≤ C | det( B ′ m ) | , and then ∀ i ∈ [ d ] , | w i | ≤ C , which implies that B ′ m is a C -approximate barycentric spanner of S ′ m .

Step 3: B m is a C -approximate barycentric spanner of S m . Here, we show that every a ∈ S m can be written as a linear combination of elements only in B m with coefficients in [ -C, C ] . Notice that we only need to consider a ∈ S m - B m since if a ∈ B m , then, one can find a trivial linear combination by itself (recall that we assume C > 1 ). Then, we prove the desired claim for all a ∈ S m -B m by contradiction. Suppose that there exists an arm a ∈ S m -B m such that it cannot be linearly represented only by elements in B m with coefficients in [ -C, C ] . As S m ⊆ S ′ m and B ′ m is a C -approximate barycentric spanner of S ′ m (shown in step 2), there must exist coefficients { d i } d i =1 to linearly represent a as:

̸

̸

As for Eq. (49), we assume without loss of generality that the coefficient index j ∈ I m satisfies d j = 0 . Since a / ∈ B m , the second for-loop in Algorithm 12 ensures that a can be represented as a linear combination of { b i } i ∈ [ d ]: i = j , and it would be put in B m otherwise (recall that Algorithm 12 terminates the second for-loop, when it scans every column vector of A and makes no replacement). Hence, there exist coefficients { c i } i ∈ [ d ]: i = j such that

̸

̸

Bridging Eq. (49) and Eq. (50), we have

̸

̸

̸

which implies that the j -th element of B ′ m , i.e., b j can be expressed as a linear combination of other elements of B ′ m , i.e., { b i } i = j . This leads to a contradiction because Algorithm 12 ensures B ′ m to be linearly independent. As a consequence, every element of a ∈ S m can be written as a linear combination of elements only in B m with coefficients in [ -C, C ] , which completes the proof.

Proof of Lemma G.1. Recall from Eq. (3) that A m is defined as

Since Lemma G.6 gives that B m is a C -approximate barycentric spanner of S m , it suffices to show that A m ⊆ S m to prove the claimed result. We prove this by induction. For the base case m = 1 , A 1 ⊆ S 1 trivially holds based on definitions. Suppose that A m ⊆ S m holds for m ≥ 2 . For m +1 , A m +1 is defined as:

Since Lemma G.6, gives that B m is a C -approximate barycentric spanner of S m , and the inductive hypothesis gives A m ⊆ S m , we have A m ⊆ Span [ -C,C ] ( B m ) , which implies that

Once the induction is done, the proof is complete.

̸

̸

G.6 Proof of Theorem 4.2: Regret Analysis

Before proving Theorem 4.2, we first present technical concept and lemmas. We first formally introduce the concept of C -approximate optimal design, both of which help us to prove a stronger result of Eq. (4), paving the way to prove the main theorem. Let ∆( A ) be the set of all Radon probability measures over set A .

Definition G.7 ( C -approximate optimal design) . Suppose that A ⊆ R d is a finite and compact set. A distribution π ∈ ∆( A ) is called a C -approximate optimal design with an approximation factor C ≥ 1 , if

Then, the following lemma shows that if one computes a uniform design for set B , a barycentric spanner for another set A , then, playing arms in B can guarantee accurate estimations over A .

Lemma G.8 (Lemma 2 of [Zhu et al., 2022]) . Suppose that A ⊆ R d is a compact set that spans R d . If B = [ b 1 , · · · , b d ] is a C -approximate barycentric spanner for A , then, π : B → 1 d is a ( C 2 · d ) -approximate optimal design, which guarantees

We then present the following lemma which provides a stronger result than that of Eq. (4). More specifically, we are supposed to show that ∥ a ∥ V -1 m ≤ C · d / √ T m holds for all a ∈ A m in Eq. (4), but we will show that this holds for all a ∈ S m . This is stronger since A m ⊆ S m for all m ∈ N .

Lemma G.9. For S m defined in Eq. (48) and B m returned by Algorithm 12, setting π m ( a ) = 1 d for each a ∈ B m and playing each arm a ∈ B m for n m ( a ) = ⌈ T m π m ( a ) ⌉ times ensure that

Proof. For all m ∈ N , we define B ′ m = B m ∪ ( ∪ i ∈I m e i √ T m ) and

With this definition and also the definition n m ( a ) = ⌈ T m π m ( a ) ⌉ , we have

where the last step follows from the fact that

Since B ′ m spans R d and B ′ m ⊆ Span [ -C,C ] ( B ′ m ) , set Span [ -C,C ] ( B ′ m ) also spans R d . Lemma G.8 gives that

Thus, one can use Eq. (52) to show that for all

As B m is a C -approximate barycentric spanner of S m , we have S m ⊆ Span [ -C,C ] ( B m ) ⊆ Span [ -C,C ] ( B ′ m ) , the proof is thus complete.

The following analysis will condition on the nice event E , defined as:

where S m is defined in Eq. (48).

Lemma G.10. We have P ( E ) ≥ 1 -δ .

Proof. One can show that

where the first inequality uses Cauchy-Schwartz inequality and the second inequality uses Lemma G.9.

Notice that the above inequality holds for all m ∈ N . Finally, Theorem 20.5 of [Lattimore and Szepesvári, 2020] shows that with probability at least 1 -δ , for all m ∈ N

Combining Eq. (55) and Eq. (54), we have that ∀ a ∈ S m ,

where the third inequality uses 4 log(4 x ) ≤ 8 x for all x ∈ R > 0 , the fourth inequality follows from a + b ≤ 2 ab whenever a, b > 1 (here a = 8 C and b = 2log( d 3 4 m /δ ) ), and the last inequality holds due to C > 1 .

Hence, the proof is complete.

Lemma G.11. Suppose E occurs where E is in Eq. (53) . For all m ∈ N , a ⋆ ∈ S m holds where S m is defined in Eq. (48) .

Proof. We prove this by induction. For the base case m = 1 , the claim holds trivially. Suppose that it holds for phase m (i.e., a ⋆ ∈ S m ) and then we aim to show a ⋆ ∈ S m +1 . For all a ∈ S m , one can show

where the second inequality follows from the definition of E (see Eq. (53)) and a ⋆ ∈ S m by inductive hypothesis. Then, we have

The inductive hypothesis gives a ⋆ ∈ S m , and thus

since Lemma G.6 gives that B m is a C -approximate barycentric spanner of S m . Combining 〈 ̂ θ m +1 , a ⋆ -a ⋆ m +1 〉 + 2 -m ≥ 0 and Eq. (56), we have a ⋆ ∈ S m +1 according to definition of S m in Eq. (48). Once the induction is done, the proof is complete.

Lemma G.12. Suppose that E occurs where E is in Eq. (53) . For each arm a ∈ A with ∆ a > 0 , it will not be in S m for all phases m ≥ m a +1 , where m a is the smallest phase such that ∆ a 2 > 2 -m a .

Proof. Consider an arbitrary arm a ∈ A with ∆ a > 0 . Let m a be the smallest phase such that ∆ a 2 > 2 -m a (i.e., ∆ a 2 ≤ 2 -( m a -1) ). Then, we will show that arm a will be not in S τ for all τ ≥ m a . Suppose that a ∈ S m a (if not, it does not impact the claim). One can show that

where the first inequality uses Lemma G.11 that a ⋆ ∈ S m for all m ∈ N , the second inequality follows from the definition of E (see Eq. (53)), and the last inequality holds due to the choice of m a .

According to the definition of S m in Eq. (48), arm a will not be in S m for all m ≥ m a +1 as long as E occurs.

Lemma G.13. Let m ( t ) be the phase in which round t lies. Then, m ( t ) ≤ log 2 ( t +1) for all t ∈ N .

Proof. We prove this by contradiction. Suppose that ∃ t ∈ N that m ( t ) > log 2 ( t +1) . Note that we can further assume m ( t ) ≥ 2 since one can easily verify that for all t such that m ( t ) = 1 , m ( t ) ≤ log 2 ( t +1) must hold. Recall that in phase m ( t ) , each active arm will be played for m ℓ ( t ) times, we have

where the third inequality bounds ℓ ( t ) in the logarithmic term by ℓ ( t ) ≥ 2 and bound the other ℓ ( t ) > log 2 ( t +1) by assumption. Therefore, once a contradiction occurs, the proof is complete.

Lemma G.14. Let m ( t ) be the phase in which round t lies. Suppose that E occurs where E is in Eq. (53) and Algorithm 2 computes a C -approximate barycentric spanner with C > 1 . For all t ∈ N and all a ∈ A , if a ∈ S m ( t ) , then,

Proof. If a ∈ S m ( t ) is optimal, then, ∆ a = 0 and the claim trivially holds. In what follows, we only consider arm a ∈ S m ( t ) with ∆ a > 0 . Consider an arbitrary round t ∈ N and an arbitrary arm a ∈ S m ( t ) . Then, t can be bounded by

where the second inequality holds because π s ( a ) = 1 d for all s ∈ N and all a ∈ B s , thereby π s ( a ) T s ≥ 1 , which gives ⌈ π s ( a ) T s ⌉ ≤ 2 π s ( a ) T s , the inequality (a) bounds m ( t ) ≤ m a where m a is defined in Lemma G.12, the inequality (b) uses ∆ a 2 ≤ 2 -m a +1 to bound m a ≤ log 2 ( 4 / ∆ a ) , and the inequality (c) follows from Lemma G.13 that m ( t ) ≤ log 2 ( t +1) .

Conditioning on E , this argument holds for each t, a , which completes the proof.

Proof of Theorem 4.2. Once Lemma G.11 and Lemma G.14 hold, Theorem 3.1 gives that for any fixed δ ∈ (0 , 1) , Algorithm 10 achieves the ULI guarantee with a function (omitting C as it is a constant)

Therefore, the proof is complete.

G.7 Proof of Theorem 4.2: Computational Analysis

As the second for-loop restarts repeatedly, and we first present the following lemma to bound the number of times that it restarts.

Lemma G.15. Under the same setting of Lemma G.1, Algorithm 12 outputs a C -barycentric spanner by restarting the second for-loop for O ( d 2 log C ( d ) ) times.

Proof. According to [Awerbuch and Kleinberg, 2008, Lemma 2.6], if I m never changes after entering the second for-loop, then, the second for-loop restarts for at most O ( d log C ( d )) times, and then the algorithm terminates. In Algorithm 12, if I m changes, the second for-loop restarts. As set I m is always non-increasing, it suffices to consider the worst case ( I m changes at most O ( d ) times). Hence, the second for-loop restarts at most O ( d 2 log C ( d )) times.

Now, we are ready to show the computational complexity.

From Lemma G.15, Algorithm 12 restarts the second for-loop at most O ( d 2 log C ( d )) times. For each run of the second for-loop, the optimization oracle is invoked at most d times. Thus, the number of calls to the oracle is at most O ( d 3 log C ( d )) . Apart from the second for-loop, the first for-loop invokes the oracle d times and computing the empirical best arm a ⋆ m requires once call to the oracle. Combining all together, we obtain the claimed bound.

H Omitted Details of Section 5

H.1 Proof of Theorem 5.1

The proof of main theorem conditions on a nice event E in which some high-probability bounds hold simultaneously. We defer the formal definition of E to Appendix H.2.

The key to conducting policy elimination is to ensure that the estimated value functions will be closer to the true value functions as phases evolve. The following proposition gives us the desired result.

Proposition H.1. Suppose that E occurs. For all m ∈ N and π ∈ Π m , we have

With the above result at hand, one can treat each policy as an arm and repeat the same arguments in Appendix D.2 (counterparts are Lemma D.7, Lemma D.8, and Lemma D.9) to get the following three lemmas.

Lemma H.2. Suppose that E occurs. For each m ∈ N , π ⋆ ∈ Π m holds.

Lemma H.3. Suppose that E occurs. For each policy π with ∆ π > 0 , it will not be in Π m for all phases m ≥ m π +1 where m π is the smallest phase such that ∆ π 2 > 2 -m π .

Lemma H.4. Let m ( t ) be the phase that round t lies in. Then, m ( t ) ≤ log 2 ( t +1) for all t ∈ N .

The next lemma shows that if a policy is not eliminated, then, the policy gap is in order of ˜ O ( t -1 / 2 ) .

Lemma H.5. Let m ( t ) be the phase in which episode/round t lies. Suppose that E occurs. For all t ∈ N and all π ∈ Π , if π ∈ Π m ( t ) , then

Proof. If π ∈ Π m ( t ) is optimal, then, ∆ π = 0 and the claim trivially holds. In what follows, we only consider policy π ∈ Π m ( t ) with ∆ π > 0 . From Lemma H.3, if a policy π ∈ Π m ( t ) is with ∆ π > 0 , then, m ( t ) ≤ m π where m π is defined in Lemma H.3. Thus, the total number of episodes/rounds that such a policy π is active is at most

where the second inequality holds due to s ≤ m ( t ) ≤ log 2 ( t +1) , and the last step uses the fact that Π all = A SH . Rearranging the above, we complete the proof.

Now, Theorem 5.1 is immediate if we treat each policy as an arm and invokes Theorem 3.1.

H.2 Construction of Nice Event

We extend the definitions of value function and action value functions to reward-dependent ones.

Next, we start to construct high-probability event. We first construct an event that high-probability bounds occur for a single phase (i.e., a single execution of Algorithm 4), and then extend it to the case in which those bounds simultaneously hold for all phases.

Lemma H.6. Suppose that Algorithm 4 is executed with input ( δ, Π , T ) where Π ⊆ Π all . With probability at least 1 -δ/ 5 , for all ( t, h, π, s, a ) ∈ [ T ] × [ H ] × Π ×S × A :

Proof. For ( s, a ) has been visited, we apply Hoeffding's inequality and union bounds to complete the proof. For those ( s, a ) that has not been visited yet, the claim trivially holds true.

Lemma H.7. Suppose that Algorithm 4 is executed with input ( δ, Π , T ) where Π ⊆ Π all . With probability at least 1 -δ/ 5 , for all ( h, s, a, π, t ) ∈ [ H ] ×S × A × Π × [ T ] ,

Proof. For those ( s, a ) that has not been visited yet, the claim trivially holds true. For ( s, a ) has been visited, by Bernstein's inequality, with probability at least 1 -δ ′ ,

where the last inequality uses the Cauchy-Schwarz inequality. By taking a union bound, choosing a proper δ ′ , and using the definition of b t,h ( s, a ) , we complete the proof.

In the following high-probability bounds, we again assume that Algorithm 4 is executed with input ( δ, Π , T ) where Π ⊆ Π all . By Azuma-Hoeffding's inequality and union bounds, with probability at least 1 -δ/ 5 , for all π ∈ Π ,

For shorthand, we denote

By Azuma-Hoeffding's inequality and union bounds, with probability at least 1 -δ/ 5 , for all ( t, h ) ∈ [ T ] × [ H ] ,

where c 2 > 0 is some absolute constant.

Again, by the fact that | Π all | ≥ | Π | and Azuma-Hoeffding's inequality and union bounds, with probability at least 1 -δ m / 5 , for all π ∈ Π ,

Definition H.8 (definition of ̂ E , single phase) . Suppose that Algorithm 4 is executed with input ( δ ′ , Π , T ) where Π ⊆ Π all . Let ̂ E be the event that all high probability bounds in Eq. (58) , Eq. (60) , and Eq. (61) hold simultaneously.

Taking a union bound over those high-probability bounds, we have

Definition H.9 (definition of E ) . Let E be the event that when running Algorithm 3, all high probability bounds in Eq. (58) , Eq. (60) , and Eq. (61) hold for all phases m ∈ N simultaneously.

Recall that Algorithm 3 runs Algorithm 4 in phases with input ( δ m , Π m , T m ) where δ m = δ/ (2 m 2 ) and ∀ m ∈ N , | Π m | ≤ | Π all | holds. By a union bound over all m ∈ N , P ( E ) ≥ 1 -δ .

H.3 Supporting Lemmas

Recall the reward-dependent value function given in Eq. (57), and we give the following lemmas. Lemma H.10. Suppose that Algorithm 4 is executed with input ( δ ′ , Π , T ) where Π ⊆ Π all , and suppose ̂ E occurs. For all ( π, t, h, s ) ∈ Π × [ T ] × [ H ] ×S , V π h ( s, b t /H ) ≤ ̂ V π h,t ( s ) holds.

Proof. Conditioning on ̂ E , one can use Lemma H.6 and follow the same idea of Lemma 18 in [Azar et al., 2017] to complete the proof.

Lemma H.11. Suppose that Algorithm 4 is executed with input ( δ ′ , Π , T ) where Π ⊆ Π all , and suppose ̂ E occurs. For all policy π ∈ Π , the following holds.

Proof. One can show the following:

where the second inequality follows from Lemma H.7.

Since those concentration bounds are also two-sided, the other side of desired claim can be similarly proved. Note that for the other side, one only need to consider ̂ Q π T, 1 ( s, a ) ≤ H for all ( s, a ) , and otherwise, the difference is negative, which implies that the claim trivially holds.

Lemma H.12. Suppose that Algorithm 4 is executed with input ( δ ′ , Π , T ) where Π ⊆ Π all , and suppose ̂ E occurs. For ξ t,h defined in Eq. (59) , we have

Proof. Conditioning on ̂ E , we have

where the first inequality holds due to Lemma H.7.

Then, we turn to bound two terms in Eq. (63). Note that Eq. (60) already gives the bound of the first term, and thus we only need to bound the second term. Before that, we first present an auxiliary lemma. The proof of this lemma can be found in [Jin et al., 2020, Lemma 10]

Lemma H.13. Suppose that Algorithm 4 is executed with input ( δ ′ , Π , T ) where Π ⊆ Π all . For all h ∈ [ H ] , the followings hold.

With the above lemma in hand, one can show:

where the last step holds due to Lemma H.13.

Therefore, we have

H.4 Proof of Proposition H.1

We first consider a single execution of Algorithm 4 with input ( δ, Π , T ) where Π ⊆ Π all . With the above supporting results, we are now ready to prove the claimed result. The following proof will

condition on event E . First, one can show that for all π ∈ Π :

Then, we turn to bound the second term. Recall the reward dependent value function defined in Eq. (57), and we have

From Eq. (58), we have

Combining the above with Eq. (64), we arrive at

Now, we consider a fixed phase m where the input of Algorithm 4 is ( δ m , Π m , T m ) . There always exist two absolute constants c 1 , c 2 > 0 for T m = c 1 2 -2 m S 2 AH 4 log 2 ( c 2 2 -2 m S 2 AH 4 | Π all | δ -1 ) such that

As conditioning on event E , the above holds for all m ∈ N . Thus, the proof is complete.

I Limitations

One limitation is that the regret bound implied by the ULI guarantee for linear bandits with infinitelymany arms is suboptimal, i.e., ˜ O ( d 1 . 5 √ T ) compared with lower bound Θ( d √ T ) but note that this problem is long-standing open. This is discussed below Theorem 4.2. We also mention the limitation of Algorithm 3 (below Theorem 5.1) that it is computational inefficient and the regret bound implied by ULI guarantee is also suboptimal.

J Broader Impacts

This paper proposes a new metric for online RL, called uniform last-iterate, which could be helpful for high-stacks applications, since a near-optimal result under our metric ensures both good cumulative and instantaneous results.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We itemize two main questions in introduction section and also itemize our contributions on how to approach and resolve those issues in our paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to Appendix I.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions can be found in Section 2 and all proofs can be found in appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This paper is theoretically oriented and does not conduct any experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper is theoretically oriented and does not conduct any experiment.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This paper is theoretically oriented and does not conduct any experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This paper is theoretically oriented and does not conduct any experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This paper is theoretically oriented and does not conduct any experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed NeurIPS Code of Ethics and commit to it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please refer to Appendix J.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new metric for sample efficiency in online learning called Uniform Last Iterate (ULI), show that it is stronger than existing metrics, and gives algorithms that achieve near-optimal ULI.\n\nSTRENGTHS:\n1. The paper proposes a new metric for online learning that characterizes not only the cumulative performance but also the instantaneous performance of the learner, which might be important in high-risk fields. The authors show that the proposed metric, called ULI, captures instantaneous performance and is strictly stronger than existing metrics: regret and uniform-PAC. This shows that their new metric is indeed valuable.\n2. The authors then present algorithms that achieve near-optimal ULI in multi-armed bandit, linear bandits and tabular MDPs. The main insight is that algorithms based on action elimination achieve near-optimal ULI, while optimistic algorithms are proved to not achieve near-optimal ULI. This separation between optimism and action elimination is an interesting side effect.\n3. The paper is very clearly written. Presenting a new concept is not easy, and the authors do a good job in explaining it, its motivation and its differences from existing metrics. The algorithms and proofs in the appendix are also easy to follow.\n\nWEAKNESSES:\n1. Although I quite like the definition of this new metric, the potential impact of this paper seems low. The insight about action elimination algorithms being better than optimistic algorithms is nice, but we have seen other cases where this is true and optimistic algorithms are not used in practice anyways. Unfortunately, I believe that the paper cannot get a higher score because of this reason. I would be happy to hear what the authors believe could be the future works that build on this paper.\n2. While the paper and the appendix are well-written, I think that the structure of the paper could greatly improve so that the important things appear in the main text. There are many theorems but not one proof sketch. This is a shame because many of the claims are nice and simple to explain even in just two sentences. I think this would give much better intuition to the readers regarding the logic behind this new metric. In contrast, using a full page to describe an RL algorithm which is mostly standard seems like a huge waste of space. I do not feel like I gained any insight from the RL section (section 5), after reading the previous sections on bandits.\n3. The RL algorithm is disappointing because it is not computationally efficient. For me, this means that the authors did not prove that near-optimal ULI can be achieved in MDPs. The result is over claiming of the contributions in the introduction, especially since near-optimal computationally efficient algorithms based on action elimination already exist in tabular MDPs. Is there a reason they do not achieve near-optimal ULI?\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors study algorithms with better performance metrics for both bandits and reinforcement learning. They propose a new metric, namely the uniform last-iterate guarantee, generalizing uniform-PAC, which can further ensure the last-iterate performance of the algorithm. The authors present algorithms achieving this guarantee in different scenarios. In the multi-armed bandits setting, the authors propose an elimination-based meta algorithm that can achieve a near-optimal uniform last-iterate guarantee. Together with the adaptive barycentric spanner technique, they then show that the algorithm can be extended to linear bandits with possibly infinite arms. Finally, the authors generalize their algorithm to the tabular Markov decision process, achieving a uniform last-iterate guarantee with near-optimal factors.\n\nSTRENGTHS:\n- The paper is generally well-written and easy to follow.\n\n- The paper discusses a new metric for online learning algorithms. It is shown that the new uniform last-iterate guarantee is strictly stronger than the uniform-PAC guarantee, which I think is an interesting concept for study.\n\n- The paper presents a meta-algorithm that translates the uniform last-iterate guarantee into adversarial bandits. This adaptation highlights significant implications for the new metric, which are compelling and merit further exploration.\n\nWEAKNESSES:\n- The reinforcement learning algorithm presented in the paper have to examine every policy, which is generally not computationally efficient.\n\nQUESTIONS:\n- Could the authors discuss potential barriers that might prevent the phase-based algorithm (He et al., 2021) from achieving a uniform last-iterate guarantee in linear bandits?\n\nHe, Jiafan, Dongruo Zhou, and Quanquan Gu. \"Uniform-pac bounds for reinforcement learning with linear function approximation.\" Advances in Neural Information Processing Systems 34 (2021): 14188-14199.\n\nLIMITATIONS:\n- The authors have thoroughly discussed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a stronger metric, uniform last-iterate (ULI) guarantee. \n\nThe authors demonstrate that a near-optimal ULI guarantee directly implies near-optimal cumulative performance across traditional metrics such as regret and PAC-bounds, but not the other way around. \n\nThe authors first provide two results for bandit problems with finite arms, showing that some algorithms can attain near-optimal ULI guarantees. \n\nThey also provide a negative result, indicating that optimistic algorithms cannot achieve a near-optimal ULI guarantee. \n\nFinally, they propose other interesting algorithms that achieve a near-optimal ULI guarantee for linear bandits with infinitely many arms (given access to an optimization oracle) and the online reinforcement learning setting.\n\nSTRENGTHS:\n- Interesting new metric: Uniform last-iterate, which is a stricter version of uniform-PAC. \n\n- Interesting inclusion result: \n    - ULI includes uniform-PAC but the converse is not true. \n    - Also, any optimism-based algorithms cannot satisfy the ULI standard. \n    - ULI algorithms can still achieve tight minimax regret for many bandit instances (Section 3.1)\n\nWEAKNESSES:\n- Necessity of the ULI itself: basically this notation forces the learner to choose less explorative options. However, in my intuition, the only way to do less exploration later is to choose more sub-optimal arms often at the early stage where ULI is less restrictive. In your patient example in the Introduction section, does it make any major ethical difference?\n    - As somewhat expected, an algorithm with ULI standard was not able to achieve the minimax optimal regret on the general linear bandit (with infinitely many arms) since the criterion prevents the algorithm from exploring, while in linear bandit it is important to explore each direction enough. Still, admitting the weakness shows honesty in their report. \n\n- Clarity: As a first reader, it is not easy for me to understand why the $\\log (t)$ term in the optimism-based algorithms guarantees that they fail the ULI standard. They said the log term makes the algorithm choose sub-optimal arms infinitely often, but does that directly induce that it does not satisfy the ULI condition? It would be great if authors could add a theorem for it, besides the lil'UCB result. \n\n- They cannot achieve anytime sublinear expected regret, since their result is stricter than uniform-PAC.\n\nQUESTIONS:\n- What about the instance-dependent bound? UCB is strong since it also guarantees good instance-dependent bound. I guess most of the results in this paper are written in a minimax manner. It would be great if authors could provide the followings\n    - Result of instance-dependent regret, such as the counterpart of Thm 2.6\n    - Example of algorithms with optimal instance-dependent regret and satisfying ULI condition. If this is difficult, it would be great if authors could make an instance-dependent analysis for the algorithms in section 3.1. \n\n- I want to ask for additional theorems about the impossibility of optimism-based algorithms, see the weakness section above. \n\n- Necessity of the ULI: check the weakness section above. I somewhat feel ULI as 'unnecessarily strict' criterion.\n\nLIMITATIONS:\nThe authors state their limitation in their paper. \n\nNo negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new form of guarantee for MAB and RL algorithms named \"Uniform Last Iterate (ULI)\". Just like the uniform PAC guarantee introduced in Dann Lattimore and Brunskill (2017), it unifies the traditional sublinear-regret (with optimal rates) and PAC guarantees for such algorithms. However, unlike uniform-PAC, ULI guarantees are also shown to encapsulate the instantaneous perfromance of MAB/RL algorithms, ensuring that, with high probability, a very suboptimal arm/action is _never_ taken after large enough $t$ (with probability $\\geq 1 - \\delta$, the maximum suboptimality after round $t$ is bounded by a decreasing function of $t$).\n\n* They show that ULI is strictly stronger than uniform PAC (and hence also unifies regret and PAC).\n* They show that successive elimination (SE) and phased elimination (PE) algorithms achieve ULI for finite-arm and linear bandits (in the case of PE).\n* They show that well-known optimistic algorithms for stochastic bandits do not achieve ULI, and that even some modifications (lil'UCB) do not achieve near-optimal-rate ULI.\n* They show that a phased-meta algorithm used with well-known adversarial bandit algorithms such as EXP3.P can achieve ULI (Appendix F).\n* They show that phased-policy-elimination with a modified version of UCB-VI can achieve ULI for tabular episodic MDPs.\n* All the ULI guarantees are near-optimal (the ULI bound decays as $t^{-1/2}$) which leads to near optimal regret bounds $\\tilde{O}(\\sqrt{T})$,\n\nSTRENGTHS:\n* The ULI guarantee is a novel and interesting extension of existing work which unifies a number of known-guarantees applied to online bandit/RL algorithms (regret, PAC, uniform PAC).\n* ULI encapsulates instantaneous performance in a way that existing guarantees do not, which is of interest in safety-critical applications.\n* The ULI guarantee is very _applicable_ in the sense that existing well-known algorithms achieve it with mild conditions/modifications, and the authors successfully demonstrate this in the paper.\n\nWEAKNESSES:\n* The regret bounds obtained by ULI have the optimal dependency on $T$ (i.e. $\\sqrt{T}$) but suboptimal dependence on other parameters (such as $d$ for linear bandits).\n* The algorithm for episodic tabular RL is computationally inefficient (both limitations are acknowledged and drawn attention to by the authors in the paper, and are fairly minor considering the contribution).\n* There is no experimental evaluation for the modified algorithms compared to the baseline algorithms.\n\nQUESTIONS:\n* Can you comment on where the UBEV algorithm of (Dann et al 2017) with the uniform PAC guarantee fails w.r.t the ULI guarantee (ULI even if not with near optimal rates)?\n\nLIMITATIONS:\nThe limitations have been addressed adequately by the authors. No negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}