Title: Optimal Top-Two Method for Best Arm Identification and Fluid Analysis
Venue: NeurIPS 2024
Accepted: yes
Keywords: Stochastic multi-armed bandits

best-arm identification

sequential learning

ranking and selection
Abstract: Top-2 methods have become popular in solving the best arm identification (BAI) problem. The best arm, or the arm with the largest mean amongst finitely many, is identified through an algorithm that at any sequential step independently pulls the empirical best arm, with a fixed probability $\beta$, and pulls the best challenger arm otherwise. The probability of incorrect selection is guaranteed to lie below a specified $\delta>0$. Information theoretic lower bounds on sample complexity are well known for BAI problem and are matched asymptotically as $\delta\to 0$ by computationally demanding plug-in methods. The above top 2 algorithm for any $\beta\in(0, 1)$ has sample complexity within a constant of the lower bound. However, determining the optimal β that matches the lower bound has proven difficult. In this paper, we address this and propose an optimal top-2 type algorithm. We consider a function of allocations anchored at a threshold. If it exceeds the threshold then the algorithm samples the empirical best arm. Otherwise, it samples the challenger arm. We show that the proposed algorithm is optimal as $\delta\to 0$. Our analysis relies on identifying a limiting fluid dynamics of allocations that satisfy a series of ordinary differential equations pasted together and that describe the asymptotic path followed by our algorithm. We rely on the implicit function theorem to show existence and uniqueness of these fluid ode’s and to show that the proposed algorithm remains close to the ode solution.

Optimal Top-Two Method for Best Arm Identification and Fluid Analysis

Abstract

Top2 methods have become popular in solving the best arm identification (BAI) problem. The best arm, or the arm with the largest mean amongst finitely many, is identified through an algorithm that at any sequential step independently pulls the empirical best arm, with a fixed probability β , and pulls the best challenger arm otherwise. The probability of incorrect selection is guaranteed to lie below a specified δ > 0 . Information theoretic lower bounds on sample complexity are well known for BAI problem and are matched asymptotically as δ → 0 by computationally demanding plug-in methods. The above top 2 algorithm for any β ∈ (0 , 1) has sample complexity within a constant of the lower bound. However, determining the optimal β that matches the lower bound has proven difficult. In this paper, we address this and propose an optimal top-2 type algorithm. We consider a function of allocations anchored at a threshold. If it exceeds the threshold then the algorithm samples the empirical best arm. Otherwise, it samples the challenger arm. We show that the proposed algorithm is optimal as δ → 0 . Our analysis relies on identifying a limiting fluid dynamics of allocations that satisfy a series of ordinary differential equations pasted together and that describe the asymptotic path followed by our algorithm. We rely on the implicit function theorem to show existence and uniqueness of these fluid ode's and to show that the proposed algorithm remains close to the ode solution.

1 Introduction

Stochastic best arm identification (BAI) problem has attracted a great deal of attention in the multi armed bandit community (see [21], [10], [3] for some early references in BAI). The basic problem involves a finite number of unknown probability distributions or arms that can be sampled from independently and the aim is to identify the arm with the largest mean. We consider a popular fixed confidence version of the problem where the sampling is sequential and the aim is to minimise sample complexity while guaranteeing that the probability of selecting the wrong arm is restricted to a pre-specified δ > 0 . Applications are many including in healthcare and recommendation systems.

[11] developed asymptotically (as δ → 0 ) tight lower bound on sample complexity of δ -correct algorithms for these BAI problems under the assumption that arms belong to a single parameter exponential family (SPEF). This assumption reduces a probability distribution to a single parameter and allows the analysis to better focus on certain aspects of the problem structure. We retain it for similar reasons. The sample complexity lower bounds involve solving an optimization problem that also identifies optimal proportion of allocations across arms. They also propose a track-and-stop

algorithm that plugs in the empirical estimates of the distribution parameters in the lower bound and tracks the resulting approximations to optimal proportions of arms to sample. Although, this plug-in algorithm was shown to asymptotically match the lower bound, it involves repeatedly solving an optimization problem and is computationally demanding. [14] consider linear Gaussian bandits, [1] consider bandits with general distributions. Both the references propose track and stop algorithms where computation is sped up through batch processing.

Substantial literature has come up on 'top-2' based, alternative faster and intuitively appealing algorithms to identify the best arm (see [25], [26] for Bayesian approaches; [24], [22], [16] for frequentist approaches). The algorithms essentially proceed by identifying at each stage an empirical winner arm, that is, an arm with the largest mean, and its closest challenger. The empirical arm is pulled with probability β , and the challenger arm with the complimentary probability. In the frequentist setting, in [16], the challenger arm is the one with the smallest 'index function'. Heuristically, this index function measures the likelihood of the challenger arm actually being the best one. The smaller the index function, more the likelihood. Further, with high probability, the index function increases with increased allocations to the corresponding arm. As is standard (see, e.g., [11], [19]), the algorithm is terminated when the generalized log-likelihood ratio (GLLR, given in Section 3) statistic exceeds a specified threshold. These algorithms are shown to be β optimal in the sense that they match the lower bound on sample complexity satisfied by algorithms that pull the best arm β fraction of times (see [15] for non-asymptotic analysis when β = 1 / 2 ). However, determining optimal β has been an open problem that has generated considerable activity and that we address in this paper.

Contributions - Algorithm: The key insight from index based top-2 algorithm is that once a sample is given to a challenger arm with the smallest index, its index function increases. The net effect is that as the algorithm progresses, the challenger arm indexes tend to come close to each other and move together. We build upon the above insight. Through the first order conditions associated with the lower bound problem, we identify a function g that equals zero under optimal allocation when the underlying arm distributions are known. We propose an anchored top-2 type algorithm where when g > 0 , the empirical winner arm is pulled and that tends to decrease g . When g < 0 , our algorithm pulls a challenger arm (arm with the smallest index function), and that typically increases g . We observe that the indexes of challenger arms that have been pulled, tend to rise up together until they catch up with arms with higher indexes. Once challenger arms associated with all the indexes have been pulled, call this the time to stability, then, since g is close to zero and indexes of all challenger arms are close together, it can be seen that the proportionate samples to the empirical winner and the remaining arms are close to the optimal proportions as per the lower bound. This continues until the GLLR statistic exceeds a threshold, roughly of order log(1 /δ ) . The time to stability can be bounded from above by a random time with finite expectation independent of δ , while the time from stability till the GLLR statistic hits a threshold scales with log(1 /δ ) with a constant that matches the lower bound.

Fluid model: Our other key contribution is to capture the above intuitive description through constructing an idealized fluid dynamics where g stays equal to zero once it touches zero and where the indexes that have been pulled, remain equal and rise together as the algorithm progresses. We further show that the resulting equations have an invertible Jacobian. Implicit function theorem (IFT) (see [20, Appendix A.6] for an introduction to IFT) then becomes an important tool in analyzing this idealized fluid system as it allows the arm allocations ( N a : a ∈ K ) to be unique functions of the overall allocation N . IFT further allows us to identify the ordinary differential equations satisfied by the derivatives N ′ a = dN a dN as the allocations N increase. The overall path till stability is constructed by pasting together the ode paths followed by arm allocations as the set of indexes that have already been pulled and are increasing together with N , meet another higher index. Once all the indexes have been pulled, our ode stabilizes so that the proportions N a /N thereafter remain constant and equal the optimal proportions as N increases. IFT further helps show that the proposed algorithm remains close to the fluid dynamics, and matches the lower bound for small δ . For completeness, in Appendix E.2, we also identify the ode paths under fluid dynamics for β top-2 algorithms. A great deal of technical analysis goes into showing that the algorithm, observed after sufficiently large amount of samples so that the sample means are close to the true means, is close to the fluid process and they both converge to the same limit.

Other related literature: [22] also develop a top-2 type algorithm for a single parameter family of distributions. There algorithm decides between the empirical best and the challenger arm based on directional change in a certain index (related to the LB) when the underlying allocation proportions

are perturbed. It is less directly connected to the first order conditions in the LB problem compared to our algorithm. Empirically, we observe that the our proposed algorithm has lower sample complexity, and is computationally substantially faster (Their algorithm can be sub-optimal. We discuss this in Appendix D.2). [7] consider an algorithm structurally similar to ours. They focus on the BAI fixed budget (FB) setting where the total number of samples are fixed and the aim is to allocate samples to minimise the probability of incorrect selection. Unlike the fixed confidence (FC) setting (the one that we consider), the FB setting requires optimizing the first argument of relative entropy functions that appear in the lower bound. In FC setting, the second argument is optimized ([7] vary the first argument). Fundamentally, this is because FB is concerned with sample allocations that control the probability of the data conducting a large deviations to arrive at an incorrect conclusion, while FC is concerned with controlling sample allocations on high probability paths and gathering enough evidence to rule out the likelihood that the observed data is a result of large deviations. Furthermore, [7] prove weaker a.s. convergence results for associated indexes although not for allocations, and since they focus on FB settings, they do not provide sample complexity bounds or probabilistic false selection guarantees. Our analysis is more nuanced and structurally detailed, and we prove that the sample complexity of the proposed algorithm is asymptotically optimal. [28] study the bestk -arm identification problem in the BAI setting with fixed confidence and bring out the structural complexities that arise in lower bound analysis when k > 1 . For k = 1 , they develop an asymptotically optimal top-2 algorithm when arm distributions are restricted to be Gaussian. [27] consider related pure exploration problems using Frank-Wolfe algorithm. Their implementation involves solving a linear program at each iteration. [17], [13], [6] provide algorithms that provide finite δ sample complexity guarantees, however they are order optimal and do not match the constant in the lower bound.

Finally, while fluid analysis is common in many settings including mean field analysis and games (e.g., [4]), stochastic approximation (e.g., [5]) and queuing theory (e.g., [8]), to the best of our knowledge little or no work exists that arrives at it through IFT.

Roadmap: In Section 2, we describe the problem and develop lower bound related analysis. The proposed algorithm and our main result, Theorem 3.1, demonstrating algorithm's efficacy are stated in Section 3 where we also develop the relevant IFT framework. Section 4 spells out the fluid dynamics associated with the algorithm. Key steps involved in proving Theorem 3.1 are outlined in Section 5. We describe the numerical experiments in Section 6. Detailed proof of all results are in the appendix.

Key limitations: The proposed algorithm extends from SPEF to bounded random variables in a straightforward manner. While we do not provide supporting analysis (this limitation is due to space constraints), our numerical results in Appendix J suggest that our algorithm improves upon existing ones even in this setting. As is standard in the bandit literature, we also assume that samples from arm distributions are independent. Further, another limitation is the assumption of stationarity of the underlying distributions. This may be true when relatively short sampling horizons are involved.

2 Problem description and lower bound

Distributional assumption: As mentioned earlier, we focus on arm distributions that belong to a known SPEF. Let S ⊂ R denote the open set of possible means of the SPEF under consideration. The details related to SPEF are reviewed in Appendix B.

Fixed confidence BAI set-up: Consider an instance with K unknown probability distributions or arms , denoted by the mean vector µ = ( µ 1 , . . . , µ K ) , where each µ i ∈ S (we refer to each µ i interchangeably as a distribution as well as its mean in the SPEF context). As is standard in the BAI framework, we assume that there is a unique arm with the largest mean. Thus, without loss of generality µ 1 > max i ≥ 2 µ i . One way to handle the case where 2 or more arms are tied for the largest mean is to look for an ϵ -best arm (an arm whose mean is within ϵ of the best arm). However, that is technically a significantly more demanding problem (see [12]). Assuming uniqueness of the best arm and focusing on the best arm identification allows us to highlight the simple fluid dynamics underlying the proposed algorithm.

Algorithm: Given an unknown bandit instance µ , we consider algorithms that sequentially generate samples - if A N denotes the arm pulled at sample N , and X N denotes the associated reward generated independently from distribution µ A N , then A N is chosen sequentially and adaptively as a function of generated ( A n , X n : n = 1 , 2 , . . . , N -1) . Further, an algorithm stops at some finite random stopping time τ and announces the best arm. δ -correct algorithm is an algorithm that, given a δ > 0 ,

̸

stops at time τ δ > 0 and outputs a best arm estimate k τ δ such that P ( τ δ < ∞ , k τ δ = 1) ≤ δ . That is, it identifies the arm with highest mean with probability at least 1 -δ . Our interest is in identifying a δ -correct algorithm that minimizes E [ τ δ ] . To this end lower bounds on sample complexity of δ -correct algorithms are established using, e.g., the data processing inequality (see, e.g., [18]). We see that

̸

with d ( ν, x ) denoting the Kullback-Leibler divergence between two distributions in S with means ν and x , and the expectation is under measure P µ associated with µ . The infimum above is solved at x ⋆ = µ 1 E [ N 1 ]+ µ a E [ N a ] E [ N 1 ]+ E [ N a ] . With this, we obtain the lower bound E [ τ δ ] ≥ T ⋆ ( µ ) log 1 2 . 4 δ , where T ⋆ ( µ ) is the reciprocal of the optimal value of a max-min problem,

̸

where x 1 ,a = ( ω 1 µ 1 + ω a µ a ) / ( ω 1 + ω a ) and Σ K denotes a simplex in K dimension.

The popular plug-in track and stop algorithm involves solving the max-min problem (1) repeatedly for optimal weights with empirical distribution plugged in for µ above. The algorithm at each stage t , generates the next sample from an arm so that the proportion of arms sampled closely match the resulting optimal weights while ensuring an adequate, sub-linear exploration (e.g., each arm gets at least √ t samples at each stage t ).

̸

Propositions 2.1 and 2.2 below are crucial for our analysis. Proposition 2.1 helps in constructing the fluid dynamics in Section 4. Proposition 2.2 provides a characterization of the unique optimal allocation ω ⋆ = ( ω ⋆ a : a ∈ [ K ]) which motivates our algorithm's sampling strategy. Before stating the two propositions, we need some notation. Let B ⊆ [ K ] / { 1 } , and B = B ∪ { 1 } . Whenever B c = ∅ , let N B c = ( N a ≥ 0 : a ∈ B c ) denote an allocation of samples to the arms in B c and we treat this as a constant in the following discussion and also in the statement of the two propositions. Wedefine the quantity N 1 , 1 depending on N B c in the following way: 1) If B c = ∅ or ∑ a ∈ B c N a = 0 , N 1 , 1 is zero. 2) Otherwise, N 1 , 1 is the value of N 1 at which ∑ a ∈ B c d ( µ 1 ,x 1 ,a ) d ( µ a ,x 1 ,a ) = 1 for the given allocation N B c . To see existence of such N 1 , 1 , observe that whenever B c = ∅ and ∑ a ∈ B c N a > 0 , the function N 1 → ∑ a ∈ B c d ( µ 1 ,x 1 ,a ) d ( µ a ,x 1 ,a ) is continuous and it monotonically decreases from ∞ to 0 as N 1 increases from 0 to ∞ . Hence, a unique N 1 exists where this function equals 1 . We define the quantity N min = N 1 , 1 + ∑ a ∈ B c N a .

Proposition 2.1. For every positive N satisfying N ≥ N min , there is a unique set of variables N B ( N ) = ( N a ( N ) : a ∈ B ) and I B ( N ) satisfying the following conditions

̸

Furthermore, N B ( · ) and I B ( · ) are continuously differentiable w.r.t. N for N > N min .

Proposition 2.2. Upon taking B = [ K ] / { 1 } and N = 1 , N B (1) , as defined in Proposition 2.1 is same as the unique allocation ω ⋆ solving the max-min problem in (1). Further, I B (1) = T ⋆ ( µ ) -1 . Moreover, for every N > 0 , if B = [ K ] / { 1 } , the unique solution N B ( N ) = ( N a ( N ) : a ∈ [ K ]) satisfies N a ( N ) = Nω ⋆ a .

Proposition 2.1 is proved by applying the Implicit function theorem (IFT). Proposition 2.2 is subsumed by [11, Theorem 5], but we prove it using a different set of tools by applying the IFT. See Appendix D for the detailed arguments.

̸

For two vectors ν = ( ν a ∈ S : a ∈ [ K ]) and N = ( N a ∈ R ≥ 0 : a ∈ [ K ]) define the anchor function , g ( ν , N ) = ∑ a ∈ [ K ] / { ˆ j } d ( ν max ,z a ) d ( ν a ,z a ) -1 , where ˆ j = arg max a ν a , ν max = max a ν a , and z a = ( N ˆ j ν max + N a ν a ) / ( N ˆ j + N a ) for all a = ˆ j .

Remark 2.1. It follows from Proposition 2.2 that the anchor function g ( µ , ω ) = 0 and all the indexes ω 1 d ( µ 1 , x 1 ,a ) + ω a d ( µ a , x 1 ,a ) equal to each other, uniquely identify the optimal proportion ω ⋆ solving the max-min problem (1) (see Appendix D.1 for an easier and more insightful derivation

of these conditions). The algorithm proposed in Section 3 ensures that the empirical version of the anchor function g ( · ) quickly becomes close to zero and thereafter remains close to zero. Further, the indexes sequentially come close to each other and once they are close, they stay close through the remaining steps of the algorithm.

3 Anchored Top-2 (AT2) Algorithm

Notation: Some notation is needed to help state the proposed algorithm. For every arm a ∈ [ K ] and iteration N , ˜ N a ( N ) denotes the number of times arm a has been drawn till iteration N , and ˜ N ( N ) = ( ˜ N a ( N ) : a ∈ [ K ]) . Thus, N = ∑ a ˜ N a ( N ) . Let ˜ µ ( N ) = ( ˜ µ a ( N ) : a ∈ [ K ]) where ˜ µ a ( N ) denotes the sample mean of arm a at time N , i.e., ˜ µ a ( N ) = ∑ N t =1 I ( A t = a ) · X t / ˜ N a ( N ) , and ˆ i N = arg max a ∈ [ K ] ˜ µ a ( N ) , with an arbitrary tie breaking rule.

For every pair of arms a, b , define

̸

Let, I a,b ( N ) = ˜ N a ( N ) · d ( µ a , x a,b ( N )) + ˜ N b ( N ) · d ( µ b , x a,b ( N )) , and I a,b ( N ) = ˜ N a ( N ) · d ( ˜ µ a ( N ) , ˜ x a,b ( N )) + ˜ N b ( N ) · d ( ˜ µ b ( N ) , ˜ x a,b ( N )) . For a = ˆ i N , we call I ˆ i N ,a ( N ) , and I ˆ i N ,a ( N ) , respectively, actual index (or, simply index ) and empirical index of arm a at iteration N , and denote them using I a ( N ) , and I a ( N ) . For notational simplicity, we hide the dependency on N whenever it doesn't cause confusion. Note that I a ( N ) is a function of ˜ N ˆ i N ( N ) , ˜ N a ( N ) , ˜ µ ˆ i N ( N ) and ˜ µ a ( N ) .

Stopping Rule: As is typical in this literature, in our algorithm below, we follow a generalized log likelihood ratio (GLLR) to decide when to stop the algorithm. It is easy to check that min a ∈ [ K ] / { ˆ i } I a ( N ) denotes the GLLR, that is log of likelihood function (LF) evaluated at maximum likelihood estimator (MLE) divided by the LF evaluated at MLE of parameters restricted to alternate set with a different best arm compared to MLE (see [11, Section 3.2] for a detailed derivation). Define stopping time τ δ = inf { N | for all a ∈ [ K ] / { ˆ i } , I a ( N ) > β ( N,δ ) } , for an appropriate choice of threshold β ( N,δ ) . After stopping at τ δ , the algorithm outputs ˆ i τ δ as the best arm. [19, Eq. 25, Section 5.1] argued that for instances in SPEF, upon choosing β ( N,δ ) ≈ log(( K -1) /δ ) + 6 log (log( N/ 2) + 1) + 8 log(1 + 2 log(( K -1) /δ )) , the GLLR based stopping rule is δ -correct for any sampling strategy including the one we propose. In our numerical experiments, we follow [11] and choose a smaller threshold, β ( N,δ ) = log((1 + log N ) /δ ) .

Description of the AT2 and Improved AT2 (IAT2) Algorithm: The AT2 algorithm takes in confidence parameter δ > 0 and exploration parameter α ∈ (0 , 1) as inputs, and executes the following steps at iteration N :

Let V N def. = { a ∈ [ K ] | ˜ N a ( N -1) < N α } be the set of under-explored arms.

̸

If V N = ∅ , choose A N = arg min a ∈ [ K ] ˜ N a ( N -1) , and go to step 5 .

Else, if g ( ˜ µ ( N -1) , ˜ N ( N -1)) > 0 , choose the empirically best arm i.e. A N = ˆ i N -1 , and go to step 5 .

Else, if g ( ˜ µ ( N -1) , ˜ N ( N -1)) ≤ 0 , choose the challenger arm i.e. A N = arg min a ∈ [ K ] / { ˆ i N -1 } I a ( N -1) using some arbitrary tie breaking rule, and go to step 5 .

Sample X N from A N and update ˜ µ ( N ) and ˜ N ( N ) using X N , A N .

If min a ∈ [ K ] / { ˆ i N } I a ( N ) > β ( N,δ ) , terminate and return ˆ i N .

Inspired from the Improved Transportation Cost Balancing (ITCB) policy for selecting the challenger arm in [16], Improved AT2 (IAT2) algorithm has the same input and follows the same strategy for exploration (step 1 and 2) and choosing the best arm (step 3) as AT2. IAT2 differs from AT2 only by its choice of the challenger arm in step 4, where IAT2 samples from the arm A N = arg min a ∈ [ K ] / { ˆ i N -1 } ( I a ( N -1) + log ˜ N a ( N -1)) .

Empirically, we see that typically IAT2 performs better than AT2 with respect to sample complexity. In the appendix, we provide pseudo-codes of AT2 and IAT2 in Algorithms 1 and 2, respectively.

3.1 Theoretical guarantees

Proposition 3.1 below shows that the allocations made by AT2 and IAT2 algorithms converge to the optimal allocations ω ⋆ w.p. 1 in P µ . For every a ∈ [ K ] we define ˜ ω a ( N ) = ˜ N a ( N ) /N , and use ˜ ω ( N ) = ( ˜ ω a ( N ) : a ∈ [ K ]) to denote the algorithm's proportion at iteration N .

Proposition 3.1 ( Convergence to optimal proportions ) . There exists a random time T stable and a constant C 1 > 0 depending on µ , α , and K , and independent of δ , such that, E µ [ T stable ] < ∞ , and for every N ≥ T stable and arm a ∈ [ K ] ,

where ϵ ( µ ) is a positive constant depending only on µ and defined in Appendix B.

Theorem 3.1 ( Asymptotic optimality of AT2 and IAT2 ) . Both AT2 and IAT2 are δ -correct over instances in S , and are asymptotically optimal, i.e., for both the algorithms, the corresponding stopping times satisfy, lim sup δ → 0 E µ [ τ δ ] log(1 /δ ) ≤ T ⋆ ( µ ) , and lim sup δ → 0 τ δ log(1 /δ ) ≤ T ⋆ ( µ ) a.s. in P µ . Moreover, we can find a constant C > 0 depending on the instance µ and α , such that,

̸

Proof idea of Theorem 3.1: Weassume Proposition 3.1 and sketch the argument by which asymptotic optimality follows from it. For N ≥ T stable , and a ∈ [ K ] , from Proposition 3.1, ˜ N a ( N ) ≈ ω ⋆ a N and ˜ µ a ( N ) ≈ µ a . As a result, from Proposition 2.2, after T stable , I a ( N ) ≈ NI [ K ] / { 1 } (1) = NT ⋆ ( µ ) -1 for every a = 1 . Therefore, if min a =1 I a ( N ) crosses β ( N,δ ) at N = τ δ , since β ( N,δ ) = log(1 /δ )+ O (log log(1 /δ )+log log( N )) , we have τ δ T ⋆ ( µ ) -1 ≈ δ → 0 log(1 /δ )+ O (log log(1 /δ )+log log( τ δ )) , which gives lim sup δ → 0 τ δ log(1 /δ ) ≤ T ⋆ ( µ ) a.s. in P µ . Detailed proof is in Appendix H.

̸

We outline the key steps of the proof of Proposition 3.1 for AT2 in Section 5, and the detailed proof is in Appendix G.2. Similar arguments hold for IAT2. Considerable technical effort goes in proving this proposition due to the noise in the empirical estimate ˜ µ ( N ) , resulting in noise in the anchor function and the empirical indexes. However, before presenting the proof sketch, in the next section, we first observe the algorithm's dynamics in the limiting fluid regime where this noise is zero. Several of the important proof steps for the algorithmic allocations rely on insights from the simpler fluid model.

4 Fluid dynamics

Motivation: The fluid dynamics idealizes our algorithm's evolution through making assumptions at each iteration N that hold for the algorithm in the limit as the number of samples increase to infinity. Unlike the real setting with discrete samples, here we treat samples as a continuous object getting distributed between different arms as the sampling budget (also referred to as 'time') evolves. We denote the no. of samples allocated to an arm a ∈ [ K ] at some time N > 0 using N a ( N ) , and define the tuple N ( N ) = ( N a ( N ) : a ∈ [ K ]) . Note that ∑ a ∈ [ K ] N a ( N ) = N . The rate N ′ a ( N ) at which samples get allocated to arm a at time N depends on a continuous version of the AT2 algorithm, which we refer to as the algorithm's fluid dynamics. We define the index of arm a = 1 at time N as, I a ( N ) = N 1 ( N ) · d ( µ 1 , x 1 ,a ( N )) + N a ( N ) · d ( µ a , x 1 ,a ( N )) , where x 1 ,a ( N ) = N 1 ( N ) µ 1 + N a ( N ) µ a N 1 ( N )+ N a ( N ) . Notice that I a ( N ) defined in Section 3 is the index of arm a with respect to the algorithm's allocation ˜ N ( N ) , whereas in our current context, I a ( N ) represents the index with respect to the fluid allocations N ( N ) .

̸

Description of the fluid dynamics: First we explain the fluid dynamics in words. We formally characterize the fluid allocation N ( N ) via a system of ODEs in Theorem 4.1. Later in this section, we exploit the obtained ODEs to argue that, after starting the fluid dynamics from some time N 0 > 0 , the allocations N ( N ) reach the optimal proportions ω ⋆ = ( ω ⋆ a : a ∈ [ K ]) by a time

atmost ( min a ∈ [ K ] ω ⋆ a ) -1 · N 0 . In other words, for N ≥ ( min a ∈ [ K ] ω ⋆ a ) -1 · N 0 , we have N a ( N ) = ω ⋆ a · N for every arm a ∈ [ K ] irrespective of the initial allocation we had at time N 0 .

For notational simplicity, we hide the dependency on N , whenever it doesn't cause any confusion. Recall the anchor function g ( · ) introduced in Section 2. We use g to denote g ( µ , N ( N )) . For every subset A ⊆ [ K ] / { 1 } , we use A to denote A ∪ { 1 } .

̸

We start the fluid dynamics at time N 0 > 0 with some initial allocation N 0 = ( N 0 a ≥ 0 : a ∈ [ K ]) . We assume that the vector of true means µ is known. The fluid dynamics evolves according to the following steps at a given total allocation N ≥ N 0 : 1) If g > 0 , then N 1 increases with N while other N a 's for a = 1 are held constant till g = 0 ( g can be seen to be a monotonically decreasing function of N 1 ). 2) If g = 0 , let B denote the set of minimum indexes. Thus, I a ( N ) are equal for all a ∈ B (the equal value is denoted by I B ( N ) ) and I a ( N ) > I B ( N ) for all a ∈ B c . Then, as N increases, allocations N 1 and ( N a : a ∈ B ) increase such that g remains equal to zero, while the indexes in B remain equal. In Proposition 2.1, we characterize and prove existence of such allocations, which the fluid dynamics will track. Later we observe that, I B increases atleast at a linear rate and indexes of arms in B c stay bounded from above by a constant. 3) If g < 0 , let B be the set of minimum index arms and I B be the index of arms in B . In this situation, ( N a : a ∈ B ) increase with N keeping index of the arms in B equal, while N 1 and ( N a : a ∈ B c ) are unchanged. With this g also increases, since g is a strictly increasing function of N a for every a ∈ B . The dynamics in this case are simple and described in Proposition E.1 of Appendix E. 4) Once, g = 0 , and B = { 2 , . . . , K } , we show that each allocation increases linearly with N such that N ′ a = ω ⋆ a .

̸

The fluid ODEs: In Appendix E, we argue that if the fluid dynamics has g = 0 at time N 0 , then g becomes zero within a finite time by following step 1 or step 3 of the description. This is easy to observe when g > 0 at N 0 , because g is strictly decreasing in N 1 , and g → -1 as N 1 → ∞ . Therefore, following step 1, g becomes 0 at some finite N . We now consider the situation where g = 0 at some N > N 0 . Setting B to the set of minimum index arms, the algorithm evolves by tracking the allocation N B ( N ) = ( N a ( N ) : a ∈ B ) defined through the system (2) in Proposition 2.1. By Proposition 2.1, N B ( N ) is continuously differentiable w.r.t. N . Applying IFT to (2), we obtain the ODEs via which the allocations and the indexes evolve and present them in Theorem 4.1.

Some definitions: Let f ( µ , a, N ) = -∂ ∂x ( d ( µ 1 ,x ) d ( µ a ,x ) )∣ ∣ ∣ x = x 1 ,a . f ( µ , a, N ) is strictly positive because d ( µ 1 ,x ) d ( µ a ,x ) is strictly decreasing with x for x ∈ ( µ a , µ 1 ) .

Let ∆ a = µ 1 -µ a , and h a ( µ , N 1 , N a ) = f ( µ , a, N ) N 2 1 ∆ a ( N 1 + N a ) 2 . For notational simplicity, we denote h ( µ , N 1 , N a ) by h a . Further, for each a , we denote d ( µ 1 , x 1 ,a ) by d 1 ,a and d ( µ a , x 1 ,a ) by d a,a . Recall that for given allocations ( N a : a ∈ [ K ]) , B denotes a set such that N 1 d 1 ,a + N a d a,a = I B ( N ) for all a ∈ B , and N 1 d 1 ,a + N a d a,a > I B ( N ) for all a ∈ B c . Let h ( B ) = ∑ a ∈ B h a d -1 a,a , h ( N ) = ∑ a ∈ B c h a N a , and d B = (∑ a ∈ B d -1 a,a ) -1 .

Theorem 4.1 ( Fluid ODEs ) . If at total allocation N ≥ N 0 , we have g = 0 , and B is the set of minimum index arms, i.e., B = arg min a ∈ [ K ] / { 1 } I a ( N ) , then the following holds true:

As N increases, and till I B ( N ) increases to hit an index in B c ,

for all b ∈ B . It follows that, I ′ B ( N ) = I B ( N ) h ( B )+ h ( N ) ( N 1 + ∑ a ∈ B N a ) h ( B )+ d -1 B h ( N )

Furthermore, for a ∈ B c , I ′ a ( N ) = N ′ 1 d 1 a = N 1 h ( B ) d 1 a ( N 1 + ∑ a ∈ B N a ) h ( B )+ d -1 B h ( N ) .

̸

There exists a β > 0 , independent of N such that I ′ B ( N ) > β . In addition, for a ∈ B c , N ′ a = 0 , I a ( N ) ≤ N 0 a d ( µ a , µ 1 ) , thus the index is bounded from above. Thus, if B c = ∅ , I B ( N ) eventually catches up with another index in B c . In this way, the set B grows into { 2 , . . . , K } .

Indexes once they meet must stay together: In Appendix F.1 we argue via contradiction that in our fluid dynamics, once a set of smallest indexes that are equal, increase and catch up with another index, their union then remains equal and increases together with N . This argument is important as it motivates the proof in our algorithm that after sufficient amount of samples, once a sub-optimal arm is pulled, its index stays close to indexes of the other arms that have been pulled.

̸

Bounding the time to reach optimal proportion: We define N ⋆ to be smallest time after N 0 at which the fluid dynamics has both B = { 2 , . . . , K } and g = 0 . Let ( N ⋆ a : a ∈ [ K ]) be the allocation at N ⋆ . We first argue that: there exists i ∈ [ K ] such that N ⋆ i = N 0 i . We have argued before that if g = 0 at N 0 , then g becomes zero by some finite time, which we call M . By definition M ≤ N ⋆ . Now if B = { 2 , . . . , K } at M or M = N 0 , then after time M the fluid dynamics evolve by the ODEs in (3) and N ⋆ is the time at which B becomes { 2 , . . . , K } , which is finite by statement 3 of Theorem 4.1. In this case i is the last element to be added to B . Otherwise if B = { 2 , . . . , K } at M and M > N 0 , the only way this can happen is g < 0 in [ N 0 , M ) . In this case, i = 1 and M = N ⋆ . Since g = 0 and B = { 2 , . . . , K } at time N ⋆ , Proposition 2.2 implies N ⋆ a = ω ⋆ a N ⋆ for all a . Combining our observations, we have ω ⋆ i N ⋆ = N ⋆ i = N 0 i ≤ N 0 . Hence N ⋆ ≤ N 0 ω ⋆ i ≤ (min a ∈ [ K ] ω ⋆ a ) -1 N 0 .

̸

Thus N ⋆ is within a constant times of N 0 . We bound T stable of Proposition 5.1 using a similar argument.

Remark 4.1 ( Incorporating the stopping rule into the fluid dynamics ) . At stopping time the idealized GLLR (which is the GLLR defined in Section 3 by replacing the estimated means with the true means) just exceeds log(1 /δ ) . Since the idealized GLLR grows linearly with the allocated samples, the stopping time increases linearly with log(1 /δ ) . Since the time for fluid dynamics to reach stability is independent of δ , for small δ , stability will be reached before the algorithm stops.

Remark 4.2 ( β -fluid dynamics ) . In Appendix E.2, we construct the fluid dynamics for the β -EBTCB algorithm [16] using IFT. We prove that, for every β ∈ (0 , 1) , the β -fluid dynamics started at some time N 0 > 0 reach the β -optimal proportion (which is the solution to the max-min problem 1 with the added constraint ω 1 = β ) by a time which is a constant times N 0 .

5 Convergence of algorithmic allocations to the optimal proportions

We now outline the proof steps for Proposition 3.1. To simplify our analysis, we analyze the AT2 algorithm after the random time T 0 defined as,

̸

after which the estimates ˜ µ ( N ) are converging to µ . Recall that α ∈ (0 , 1) is the exploration parameter, and ϵ ( µ ) > 0 is a constant depending only on µ . By the definition of ϵ ( µ ) in Appendix B, we have ˜ µ a ( N ) < ˜ µ 1 ( N ) for all a = 1 and N ≥ T 0 . As a result, arm 1 becomes the empirically best arm after T 0 . In Appendix G.3, we use Chernoff's bound to prove that P µ ( T 0 = n + 1) = exp( -Ω ( n α/ 4 )) , which implies E µ [ T 0 ] < ∞ . In the following discussion, all the results mentioned are true for both AT2 and IAT2 algorithms.

Proposition 5.1 shows that the allocations made by the proposed algorithm converges to the first order condition satisfied by the optimal proportion ω ⋆ = ( ω ⋆ a : a ∈ [ K ]) at a rate O ( N -3 α/ 8 ) , where α is the exploration parameter.

Proposition 5.1. There exists a random time T stable ≥ T 0 satisfying E µ [ T stable ] < ∞ and a constant C 2 > 0 depending on µ , α and K , and independent of the sample paths, such that, for N ≥ T stable

̸

Before outlining the proof of Proposition 5.1, we explain how Proposition 3.1 follows from Proposition 5.1 just using the IFT.

Proof idea of Proposition 3.1: If our algorithm follows optimal proportions at time N , i.e. , ˜ N a ( N ) = ω ⋆ a N for all a ∈ [ K ] , RHS of (4) and (5) becomes zero by Proposition 2.2. Moreover, by Proposition 2.2 ω ⋆ uniquely satisfies the conditions: anchor function is zero and all alternative arms have equal index. (4) and (5) imply that, ˜ ω ( N ) satisfies these conditions upto a perturbation of C 2 N -3 α/ 8 after T stable . Using the IFT, we prove that the algorithm's allocation is a Lipschitz continuous function of the perturbation when it is sufficiently small. Hence, by choosing T stable large enough and using Lipschitzness, we get max a ∈ [ K ] | ˜ ω a ( N ) -ω ⋆ a | = O ( N -3 α/ 8 ) . Closeness of ˜ µ ( · ) to µ follows from the fact that T stable ≥ T 0 .

Proof idea of Proposition 5.1: We separately outline the proofs of (4) and (5) in Proposition 5.1. In the following discussion, constants hidden in O ( · ) , Ω ( · ) and Θ ( · ) notations are independent of the sample path after time T stable . To simplify our analysis, we choose T stable such that exploration stops after T stable , i.e. , V N = ∅ for N ≥ T stable (see the discussion before Definition G.1 in Appendix G.1.1 for justification).

Key ideas in the proof of (4) : We prove (4) via induction. We prove the existence of a constant D > 0 such that at every N ≥ T stable , whenever the actual anchor value g ( µ , ˜ N ( N )) (we denote using g ( N ) ) satisfies | g ( N ) | > DN -3 α/ 8 , our algorithm pushes g ( · ) towards zero by Θ (1 /N ) in the next iteration through steps 3 and 4. Whereas the interval [ -C 2 N -3 α/ 8 , C 2 N -3 α/ 8 ] shrinks by O ( N -(1+3 α/ 8) ) from both ends. Since N -(1+3 α/ 8) << N -1 , we choose the constant C 2 large enough such that g ( · ) stays in the said interval in iteration N +1 .

Key ideas in the proof of (5) : The following lemma forms a crucial part of the argument for proving closeness of the indexes in the non-fluid setting.

Lemma 5.1. There exists a random time T good ∈ [ T 0 , T stable ] such that the algorithm picks all the alternative arms in [ K ] / { 1 } atleast once between the iterations T good and T stable . Moreover, for N ≥ T good , if the algorithm picks some arm a ∈ [ K ] / { 1 } at iteration N , then it picks arm a again within a next O ( N 1 -3 α/ 8 ) iterations.

̸

Proof of Lemma 5.1 (in Appendix G.1.2) is technically involved and requires proving several supplementary lemmas. Several of the key steps of this proof borrow insights from the fluid dynamics, and we outline them in Appendix F. Here we assume Lemma 5.1 and sketch the argument by which (5) follows from it for the AT2 algorithm. For any a, b = 1 and after any N ≥ T stable , I a ( · ) and I b ( · ) crosses each other before the algorithm picks both a, b atleast once. We can show that, for j = a, b and N ≥ T stable , I j ( N ) and I j ( N ) differ by O ( N 1 -3 α/ 8 ) . As a result, when I a ( · ) crosses I b ( · ) at N + R , we have | I a ( N + R ) -I b ( N + R ) | = O (( N + R ) 1 -3 α/ 8 ) = O ( N 1 -3 α/ 8 ) since R = O ( N 1 -3 α/ 8 ) . For j = a, b , the partial derivatives of I j ( · ) w.r.t. ˜ N 1 and ˜ N j are non-negative and bounded from above by max { d ( µ 1 , µ j ) , d ( µ j , µ 1 ) } = O (1) . As a result, | I j ( N + R ) -I j ( N ) | = O ( R ) = O ( N 1 -3 α/ 8 ) for j = a, b . Hence, | I a ( N ) -I b ( N ) | ≤ | I a ( N + R ) -I b ( N + R ) | + ∑ j = a,b | I j ( N + R ) -I j ( N ) | = O ( N 1 -3 α/ 8 ) .

Bounding T stable : In Appendix G.2, we choose T good and T stable such that T stable is the time after T good by which the algorithm picks all the sub-optimal arms atleast once. By Proposition 3.1, the algorithm approximately matches ω ⋆ after T stable . Using an argument similar to the one for bounding time to reach the optimal proportion in the fluid dynamics of Section 4, we can prove that T stable ⪅ ( ω ⋆ min ) -1 T good a.s. in P µ , where ω ⋆ min = min a ∈ [ K ] ω ⋆ a (Lemma G.4, Appendix G.2.1).

Role of forced exploration in analysis: As we observe in the numerical results in Appendix J.4, forced exploration (step 1 of our algorithm) does not increase the observed sample complexity. We emphasize that without the forced exploration, Propositions 5.1 and 3.1 continue to hold if we can show that the proposed algorithm perform sufficient exploration over the instance. That is, after a random time T depending on the instance and satisfying E [ T ] < ∞ , every arm has ˜ N a ( N ) = Ω ( √ N ) . As a result, upon proving sufficient exploration, asymptotic optimality will follow without the forced exploration step.

In Appendix J.4, we see in the numerical experiments, when there is no forced exploration, AT2's sample complexity blows up over instances where multiple sub-optimal arms have equal mean. On

Acknowledgments: We thank Arun Suggala and Karthikeyan Shanmugam from Google Research Bangalore for initial discussions on this project. The second and the third author initiated this work while visiting Google Research in Bangalore.

References

Shubhada Agrawal, Sandeep Juneja, and Peter Glynn. Optimal δ -correct best-arm selection for heavy-tailed distributions. In Algorithmic Learning Theory , pages 61-110. PMLR, 2020.

Shubhada Agrawal, Sandeep Juneja, Karthikeyan Shanmugam, and Arun Sai Suggala. Optimal best-arm identification in bandits with access to offline data. arXiv preprint arXiv:2306.09048 , 2023.

Jean-Yves Audibert, Sébastien Bubeck, and Rémi Munos. Best arm identification in multi-armed bandits. In Conference on Learning Theory , pages 41-53, 2010.

Alain Bensoussan, Jens Frehse, Phillip Yam, et al. Mean field games and mean field type control theory , volume 101. Springer, 2013.

Vivek S Borkar. Stochastic approximation: a dynamical systems viewpoint , volume 48. Springer, 2009.

Lijie Chen, Jian Li, and Mingda Qiao. Towards instance optimal bounds for best arm identification. In Conference on Learning Theory , pages 535-592. PMLR, 2017.

Ye Chen and Ilya O Ryzhov. Balancing optimal large deviations in sequential selection. Management Science , 69(6):3457-3473, 2023.

Jim G Dai. On positive harris recurrence of multiclass queueing networks: a unified approach via fluid limit models. The Annals of Applied Probability , 5(1):49-77, 1995.

Amir Dembo and Ofer Zeitouni. Large deviations techniques and applications , volume 38. Springer Science & Business Media, 2009.

Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research , 7(6), 2006.

Aurélien Garivier and Emilie Kaufmann. Optimal best arm identification with fixed confidence. In Conference on Learning Theory , pages 998-1027. PMLR, 2016.

Aurélien Garivier and Emilie Kaufmann. Nonasymptotic sequential tests for overlapping hypotheses applied to near-optimal arm identification in bandit models. Sequential Analysis , 40(1):61-96, 2021.

Kevin Jamieson, Matthew Malloy, Robert Nowak, and Sébastien Bubeck. lil'ucb: An optimal exploration algorithm for multi-armed bandits. In Conference on Learning Theory , pages 423-439. PMLR, 2014.

Yassir Jedra and Alexandre Proutiere. Optimal best-arm identification in linear bandits. In Advances in Neural Information Processing Systems , volume 33, pages 10007-10017, 2020.

Marc Jourdan and Rémy Degenne. Non-asymptotic analysis of a ucb-based top two algorithm. In Advances in Neural Information Processing Systems , volume 36, pages 68980-69020, 2023.

Marc Jourdan, Rémy Degenne, Dorian Baudry, Rianne de Heide, and Emilie Kaufmann. Top two algorithms revisited. Advances in Neural Information Processing Systems , 35:26791-26803, 2022.

Shivaram Kalyanakrishnan, Ambuj Tewari, Peter Auer, and Peter Stone. Pac subset selection in stochastic multi-armed bandits. In ICML , volume 12, pages 655-662, 2012.

Emilie Kaufmann. Contributions to the Optimal Solution of Several Bandit Problems . PhD thesis, Université de Lille, 2020.

Emilie Kaufmann and Wouter M. Koolen. Mixture martingales revisited with applications to sequential tests and confidence intervals. Journal of Machine Learning Research , 22(246):1-44, 2021.

David G. Luenberger and Yinyu Ye. Linear and Nonlinear Programming . Springer US, 2008.

Shie Mannor and John N Tsitsiklis. The sample complexity of exploration in the multi-armed bandit problem. Journal of Machine Learning Research , 5(Jun):623-648, 2004.

Arpan Mukherjee and Ali Tajer. Best arm identification in stochastic bandits: Beyond β -optimality. arXiv preprint arXiv:2301.03785 , 2023.

Yurii Nesterov et al. Lectures on convex optimization , volume 137. Springer, 2018.

Chao Qin, Diego Klabjan, and Daniel Russo. Improving the expected improvement algorithm. In Advances in Neural Information Processing Systems , volume 30, 2017.

Daniel Russo. Simple bayesian algorithms for best arm identification. In Conference on Learning Theory , pages 1417-1418. PMLR, 2016.

Xuedong Shang, Rianne Heide, Pierre Menard, Emilie Kaufmann, and Michal Valko. Fixedconfidence guarantees for bayesian best-arm identification. In International Conference on Artificial Intelligence and Statistics , pages 1823-1832. PMLR, 2020.

Po-An Wang, Ruo-Chun Tzeng, and Alexandre Proutiere. Fast pure exploration via frank-wolfe. Advances in Neural Information Processing Systems , 34:5810-5821, 2021.

Wei You, Chao Qin, Zihao Wang, and Shuoguang Yang. Information-directed selection for top-two algorithms. In Proceedings of Thirty Sixth Conference on Learning Theory , volume 195 of Proceedings of Machine Learning Research , pages 2850-2851. PMLR, 12-15 Jul 2023.

Appendix

A Outline

Below we provide a brief outline of the topics presented in the appendices.

Algorithm (1) and (2) are, respectively, the pseudocodes of the AT2 and IAT2 algorithms introduced in Section 3.

Appendix B: We define the single parameter exponential family (SPEF) of distributions, and prove several inequalities bounding the index function, anchor function, and the derivatives of the anchor function, which are crucial for our analysis.

Appendix C: We introduce a framework using which we apply the implicit function theorem for proving several properties related to the fluid dynamics and the algorithm's allocations.

Appendix D: We prove Propositions 2.1 and 2.2 from Section 2.

Appendix E: We provide the proofs of the results mentioned in Section 4, and also construct the fluid dynamics for the β -EB-TCB algorithm ([16]) in Appendix E.2.

̸

Appendix F: We provide a heuristic argument to show that if the minimum index meets with the index of some sub-optimal arm a = 1 following the ODEs in Theorem 4.1, then arm a must be incorporated into the set of minimum index arms. We argue via contradiction to show that, if this is not the case then index of arm a becomes strictly less than the minimum index of the arms, which implies a contradiction. Several of the key steps in the proof of Lemma 5.1 are extensions of the said argument to the non-fluid setting of the algorithm with additional terms because of the noise in the estimates.

Appendix G: We prove Proposition 3.1, 5.1, and provide detailed proofs of all the results mentioned in Section 5.

Appendix H: We provide the detailed proof of Theorem 3.1.

Appendix I: We describe a natural extension of the proposed AT2 and IAT2 algorithms to the class of distributions with support contained in [0 , 1] . We do not theoretically analyze this algorithm owing to space constraints. However we compare the proposed algorithm with existing algorithms experimentally in Appendix J.5.

Appendix J: We compare the performance of the proposed algorithms against existing algorithms through numerical experiments. We also illustrate how our algorithm follows the fluid dynamics as the no. of samples increase.

Algorithm 1: Anchored Top-Two (AT2) Algorithm

̸

Algorithm 2: Improved Anchored Top-Two (IAT2) Algorithm

̸

B Single parameter exponential family of distributions

We consider single parameter exponential family (SPEF) of distributions of the form

where ρ is a dominating measure which we assume to be degenerate, θ lies in the interior of set Θ defined below (denoted by Θ o ):

is the log-moment generating function of the measure ρ ( · ) .

For θ, ˜ θ ∈ Θ o , the KL-divergence between the measures ν θ and ν ˜ θ is,

The mean under ν θ is given by b ′ ( θ ) . Let S be the image of the set Θ o under the mapping b ′ ( · ) . Note that S is an open interval. Also, since b ′′ ( · ) > 0 in Θ o , b ′ ( · ) is strictly increasing in Θ o , and is a bijection between Θ o and S . This implies we can parameterize the distributions in the SPEF using their means as well.

Let θ µ be the unique θ satisfying b ′ ( θ ) = µ for some µ ∈ S . Clearly, θ µ is a strictly increasing function of µ . This follows since b ′ ( · ) is strictly increasing in Θ o . Additionally, all the higher derivatives of b ( · ) exist in the set Θ o (see Exercise 2.2.24 in [9]).

For µ, ˜ µ ∈ S we define d ( µ, ˜ µ ) as,

We define µ inf ∈ R ∪{-∞} and µ sup ∈ R ∪{ + ∞} , respectively, to be the infimum and supremum of the interval S . Then, S = ( µ inf , µ sup ) .

Definition B.1. For µ = ( µ 1 , µ 2 , . . . , µ K ) ∈ S K , define r min ( µ ) = min i ∈ [ K ] { min { µ i -µ inf , µ sup -µ i }} .

and

Since S is an open interval, r min ( µ ) is positive for every µ ∈ S K , and can be ∞ if both µ inf and µ sup are ∞ .

Partial derivatives of d ( µ, ˜ µ ) : For every pair µ, ˜ µ ∈ S , the partial derivatives of d ( µ, ˜ µ ) with respect to the first argument is

and that with respect to the second argument is,

Enveloping the KL-divergence: In the following discussion, we try to bound the KL-divergence d ( µ, ˜ µ ) from both sides using the squared distance | µ -˜ µ | 2 after imposing some restrictions on the choice of µ, ˜ µ ∈ S . For an instance µ ∈ S K , we define the constants ∆ min ( µ ) and ϵ ( µ ) as

We further define

Since H ( µ ) ⊂ S , all distributions with mean in H ( µ ) have positive variance. Note that b ′′ ( θ µ ) represents the variance of the distribution with mean µ . As a result, since H ( µ ) is a compact set, both σ max ( µ ) and σ min ( µ ) are positive constants.

Hence, b ( · ) is σ min ( µ ) -strongly convex and b ′ ( · ) is σ max ( µ ) -Lipschitz on the set ( b ′ ) -1 ( H ( µ )) . Thus, using [23, Theorems 2.1.5 and 2.1.10], for every θ 1 , θ 2 ∈ ( b ′ ) -1 ( H ( µ )) , we have

and hence, for µ, ˜ µ ∈ H ( µ ) ,

Bounding the partial derivatives: We now introduce bounds on the partial derivatives d 1 and d 2 introduced earlier. Consider µ, x, ˜ µ ∈ H ( µ ) such that µ > ˜ µ , and x ∈ [ ˜ µ, µ ] . Recall

Since θ ( · ) = ( b ′ ) -1 ( · ) , we have,

Similarly, since, we have,

B.1 Enveloping the anchor and index functions under noisy estimates of the rewards

For every arm a ∈ [ K ] , let ˜ µ a be an estimate of µ a satisfying | ˜ µ a -µ a | ≤ ϵ ( µ ) and ˜ µ = ( ˜ µ a ) a ∈ [ K ] . Since ϵ ( µ ) ≤ ∆ min ( µ ) 4 , the empirically best arm with respect to the estimates ˜ µ is the first arm. Also let N a be the no. of times arm a has been pulled and N = ( N a ) a ∈ [ K ] .

Enveloping the anchor function: As we introduced in Section 2, the anchor function is,

̸

where ˜ x 1 ,a = N 1 ˜ µ 1 + N a ˜ µ a N 1 + N a . Note that ˜ µ a , ˜ x 1 ,a ∈ H ( µ ) for every a ∈ [ K ] / { 1 } . Therefore, using (6), we have,

̸

Putting ˜ x 1 ,a = N 1 ˜ µ 1 + N a ˜ µ a N 1 + N a , we obtain,

̸

̸

Now ˜ µ 1 -˜ x 1 ,a = N a N 1 + N a ( ˜ µ 1 -˜ µ a ) and ˜ x 1 ,a -˜ µ a = N 1 N 1 + N a ( ˜ µ 1 -˜ µ a ) . Using these, we can say,

̸

whenever | ˜ µ a -µ a | ≤ ϵ ( µ ) for all a ∈ [ K ] . The constants hidden in the Θ ( · ) depends only on µ and obviously the choice of the SPEF family.

Enveloping the index: Following the definition of empirical index I a ( · ) in Section 3, we define the index of any alternative arm a ∈ [ K ] / { 1 } with respect to the estimates ˜ µ = ( ˜ µ a : a ∈ [ K ]) and allocation N = ( N a : a ∈ [ K ]) is,

Observe that, the empirical index I a ( N ) and index I a ( N ) introduced in Section 3 are, respectively, equivalent to the quantities W a ( ˜ µ ( N ) , ˜ N ( N )) , and W a ( µ , ˜ N ( N )) , where ˜ µ ( N ) is the empirical estimate and ˜ N ( N ) is the algorithm's allocation at iteration N .

Using (6) we have,

Putting x 1 ,a = N 1 ˜ µ 1 + N a ˜ µ a

We define ∆ max ( µ ) = max a ∈ [ K ] / { 1 } ∆ a . Since ϵ ( µ ) ≤ 1 4 ∆ min ( µ ) , we have,

̸

Using the same notation, as we used in (10), we have,

for every arm a ∈ [ K ] / { 1 } .

Enveloping the partial derivatives of anchor function with respect to N : While analyzing the AT2 and IAT2 algorithms, we need to show that the anchor function converges to zero at a uniform rate as no. of iteration goes to infinity. During this step, we need to bound the partial derivatives of the anchor function g ( µ , N ) with respect to N . Below we evaluate the partial derivatives of g ( µ , N ) with respect to N a for different arms a ∈ [ K ] .

̸

̸

where ∆ a = µ 1 -µ a and

Using (8), and (6), we have,

where the constants hidden in Θ ( · ) depend only on µ and are independent of the sample path. Therefore,

As a consequence, we have,

̸

C Framework for applying the Implicit function theorem (IFT)

In this appendix we explain a general framework using which we later apply the Implicit function theorem for the following purposes:

Constructing the fluid dynamics for the AT2 and β -EB-TCB algorithms in Appendix E.

Proving convergence of the algorithm's allocations to the optimal proportions in Appendix G.2.2.

We introduce the variables: N = ( N a ∈ R ≥ 0 : a ∈ [ K ]) , I ∈ R , η = ( η a ∈ R : a ∈ [ K ]) , and N ≥ 0 . After fixing some instance µ ∈ S K ( S is defined in Appendix B), we define the

following functions:

where

For every non-empty subset B ⊆ [ K ] / { 1 } , we define the vector valued functions,

We denote Φ [ K ] / { 1 } ( · ) just using Φ ( · ) .

In the definitions of Φ B ( · ) and ˜ Φ B ( · ) , without loss of generality, we assume that, the functions Φ a ( · ) in the tuple ( Φ a ( · )) a ∈ B are enumerated in the increasing order of a ∈ B , i.e. , if we have B = { a 1 , a 2 , . . . , a | B | } with 1 < a 1 < a 2 < . . . < a | B | , then,

Before stating the main result of this section in Lemma C.1, we define some notation that are essential for the lemma statement. For any A ⊆ [ K ] , we use the notation N A to denote the tuple of variables ( N a : a ∈ A ) . For some vector valued function G depending on N , denote the Jacobian of G ( · ) with respect to the tuple of variables N A using ∂ G ∂ N A .

For any non-empty B ⊆ [ K ] / { 1 } , we define B = B ∪ { 1 } . For every k ≥ 1 , 0 k and 1 k , respectively, refers to a k -dimensional vector with entries 0 and 1 . We define Z B to be the set of tuples ( N , I, 0 K , N ) with N ∈ R K ≥ 0 , I ∈ R , and N ∈ R > 0 , which satisfy

Lemma C.1 ( Invertibility of the Jacobian ) . For all non-empty subset B ⊆ [ K ] / { 1 } , the following statements hold true at every tuple ( N , I, 0 K , N ) in the set Z B ,

The Jacobian ∂ ˜ Φ B ∂ N B is invertible at ( N , I, 0 K ) .

We have

at ( N , I, 0 K ) .

The Jacobian ∂ Φ B ∂ ( N B ,I ) defined as,

is invertible at ( N , I, 0 K , N ) .

Proof. Statement 1: The Jacobian ∂ ˜ Φ B ∂ N B is equivalent to,

Now we observe the following properties about the sign of the entries of the above Jacobian matrix,

We have N > 0 and Φ 1 ( N , 0 K ) = 0 . This implies N 1 > 0 and max a ∈ [ K ] / { 1 } N a > 0 . As a result, using (15), we have ∂Φ 1 ∂N 1 < 0 and ∂Φ 1 ∂N a i ≥ 0 for every i ∈ { 1 , 2 , . . . , | B |} .

For i ∈ { 2 , . . . , | B | + 1 } , in the i -th row, the only non-zero entries can be the first and the i -th entry. The first entry is ∂Φ a i ∂N 1 = d ( µ 1 , x 1 ,a i ( N 1 , N a i )) ≥ 0 . The i -th entry is ∂Φ a i ∂N a i = d ( µ a i , x 1 ,a i ( N 1 , N a i )) . Since we have N 1 > 0 , using (6), we have d ( µ a i , x 1 ,a i ( N 1 , N a i )) > 0 , making the i -th entry positive.

Therefore, considering only the sign of the elements, the matrix in (16) is of the form,

where the symbols ++ , --and + implies that the corresponding entries are positive, negative and non-negative.

We now argue that a matrix of the above structure has a rank | B | +1 . To see that, by subtracting some appropriate constant times the i -th column from the first column, we can make the entries in position i ∈ { 2 , 3 , . . . , | B | +1 } in the first column zero. As a result of these transformations, since we are subtracting non-negative quantities from the first entry of the first column, that entry remains negative. The matrix we obtain after this sequence of transformations has a structure,

Clearly a matrix of the above structure has a rank | B | +1 and therefore invertible.

Statement 2: Using statement 1 of Lemma C.1, if we take v = ( ∂ ˜ Φ B ∂ N B ) -1 ∂ ˜ Φ B ∂I , then we have,

Note that, the RHS of the above linear system i.e. ∂ ˜ Φ B ∂I is a | B | +1 dimensional vector with zero in its first entry and -1 in every other entry. Using (17), v = [ v 1 , v 2 , v 3 , . . . , v | B | +1 ] T satisfies a linear system with coefficients having the following signs,

which implies,

where ++ , --and + represents quantities which are positive, negative and non-negative.

For every i ∈ { 2 , . . . , | B | + 1 } , we can eliminate v i from the first equation by subtracting some positive constant times the i -th equation from it. After eliminating v 2 , v 3 , . . . , v | B | +1 from the first equation following the mentioned procedure, we will be left with an equation of the form ( --) v 1 = (+) , implying v 1 ≤ 0 .

Now the i -th equation of the system, for i ∈ { 2 , . . . , | B | +1 } is,

We know that, ∂Φ a i -1 ∂N 1 = d ( µ 1 , x 1 ,a i -1 ) and ∂Φ a i -1 ∂N a i -1 = d ( µ a i -1 , x 1 ,a i -1 ) , where x 1 ,a = N 1 µ 1 + N a µ a N 1 + N a for every a ∈ [ K ] / { 1 } .

Now putting the derivatives in (19), we have

for every i ∈ { 2 , . . . , | B | +1 } .

Now adding both sides for i ∈ { 2 , 3 , . . . , | B | +1 } , we get,

where the last step follows from the fact that ∑ a ∈ [ K ] / { 1 } d ( µ 1 ,x 1 ,a ) d ( µ a ,x 1 ,a ) = Φ 1 ( N , 0 K ) + 1 = 1 . Taking v 1 on the LHS, we have,

Note that the LHS of the above inequality is same as 1 T v = 1 T ( ∂ ˜ Φ B ∂ N B ) -1 ∂Φ K +1 ∂I . In the RHS, since d ( µ a , x 1 ,a ) ≤ d ( µ a , µ 1 ) , we conclude the desired result.

Statement 3: We have,

We do the following determinant preserving column operation on ∂ Φ B ∂ ( N B ,I ) ,

where [ ∂ Φ B ∂ ( N B ,I ) ] : , | B | +2 and [ ∂ Φ B ∂ ( N B ,I ) ] : , 1: | B | +1 , respectively, denotes the | B | +2 -th column and left ( | B | +2) × ( | B | +1) submatrix of ∂ Φ B ∂ ( N B ,I ) .

The above column operation gives us the matrix,

which has the same determinant as ∂ Φ B ∂ ( N B ,I ) . Therefore,

Using statement 1 and 2 of Lemma C.1, both the quantities in the above product are non-zero, making the Jacobian ∂ Φ B ∂ ( N B ,I ) invertible for every tuple in Z B .

D Proofs from Section 2

Theorem D.1 is essential for proving Propositions 2.1 and 2.2 in Section 2. Before stating Theorem D.1 we state an alternative formulation of the max-min problem 1 which we call O .

̸

where N a N 1 + N a

The optimal value of the problem O is of the form T ⋆ ( µ ) log(1 / (2 . 4 δ )) , where T ⋆ ( µ ) is the reciprocal of the optimal value of (1). If N ⋆ = ( N ⋆ a : a ∈ [ K ]) is an optimal allocation solving O , then ω ⋆ a = N ⋆ a ∑ b ∈ [ K ] N ⋆ b is an optimal proportion solving (1). Similarly if ω ⋆ is an optimal proportion solving (1) then N ⋆ = ( N ⋆ a : a ∈ [ K ]) with N ⋆ a = ω ⋆ a T ⋆ ( µ ) is an optimal allocation solving O . Theorem D.1 implies uniqueness to the solution of O which also implies uniqueness of the solution of (1).

̸

Some notations are needed before stating Theorem D.1. Let B ⊂ [ K ] / 1 and B = B ∪ { 1 } . Let ν = ( ν a : a ∈ [ K ]) ∈ S K be some instance with ν 1 > max a =1 ν a . Let ( I a : a ∈ B ) each be strictly positive. If B c = ∅ then let ( N a ∈ R ≥ 0 : a ∈ B c ) be the no. of samples allocated to arms in B c . We define N 1 , 1 as zero when ∑ a ∈ B c N a = 0 or B c = ∅ . Otherwise, we take N 1 , 1 to be the unique N 1 > 0 that solves,

̸

where x 1 ,a = ν 1 N 1 + ν a N a N 1 + N a . Note that if B c = ∅ and ∑ a ∈ B c N a > 0 , there exists an a ∈ B c with N a > 0 . As a result, the LHS of (21) decreases from ∞ to 0 as N 1 increases from 0 to ∞ . This implies the existence of a unique N 1 > 0 solving (21).

̸

Set N 1 , 2 := max a ∈ B I a d ( ν 1 , ν a ) -1 .

Theorem D.1. There exists a unique solution N ⋆ 1 ≥ 0 and ( N ⋆ a ≥ 0 : a ∈ B ) satisfying

̸

Further, N ⋆ 1 ≥ max( N 1 , 1 , N 1 , 2 ) and each N ⋆ a ≥ I a d ( ν a ,ν 1 ) .

Furthermore, The optimal solution to O is uniquely characterized by the solution above with B = { 2 , . . . , K } and each I a = log(1 / (2 . 4 δ )) and constraints (20) tight, that is, indexes of all the sub-optimal arms being equal to log(1 / (2 . 4 δ )) . Further, N ⋆ 1 ≥ max a ∈ [ K ] \ 1 log(1 / (2 . 4 δ ) d ( ν 1 ,ν a ) and each N ⋆ a ≥ log(1 / (2 . 4 δ ) d ( ν a ,ν 1 ) .

Proof: First observe that every solution N ⋆ 1 and ( N ⋆ a : a ∈ B ) to the system (22) and (23) must satisfy N 1 ≥ max { N 1 , 1 , N 1 , 2 } and N ⋆ a ≥ I a d ( ν a ,ν 1 ) , for every a ∈ B .

̸

If B c = ∅ , (22) implies,

If ∑ a ∈ B c N a = 0 , then N 1 ≥ 0 = N 1 , 1 . Otherwise, we can find an a ∈ B c with N a > 0 , making ∑ a ∈ B c d ( ν 1 ,x 1 ,a ) d ( ν a ,x 1 ,a ) strictly decreasing in N 1 . As a result, by the definition of N 1 , 1 we have N ⋆ 1 ≥ N 1 , 1 .

Now, for every a ∈ B , we have

Note that RHS of the above inequality is upper bounded by min { N ⋆ 1 d ( ν 1 , ν a ) , N ⋆ a d ( ν a , ν 1 ) } . As a result, for every a ∈ B , N ⋆ a ≥ I a d ( ν a ,ν 1 ) , and N ⋆ 1 ≥ I a d ( µ 1 ,µ a ) . This further implies, N ⋆ 1 ≥ max a ∈ B I a d ( ν 1 ,ν a ) = N 1 , 2 .

Now we prove existence of a unique N ⋆ 1 and ( N ⋆ a : a ∈ B ) . For every N 1 ≥ N 1 , 2 and a ∈ B , as N a increases from 0 to ∞ , N 1 d ( ν 1 , x 1 ,a ) + N a d ( ν a , x 1 ,a ) increases monotonically from 0 to N 1 d ( ν 1 , ν a ) . Note that N 1 d ( ν 1 , ν a ) ≥ N 1 , 2 d ( ν 1 , ν a ) ≥ I a (by the definition of N 1 , 2 ). As a result, we can find a unique N a for which N 1 d ( ν 1 , x 1 ,a ) + N a d ( ν a , x 1 ,a ) = I a . For every a ∈ B , we call that unique N a as N a ( N 1 ) . Observe that, since I a > 0 , N a ( N 1 ) is strictly decreasing in N 1 , and if I a = 0 , then N a ( N 1 ) = 0 . Also, if a = arg max b ∈ B I b d ( ν 1 ,ν b ) , then N a ( N 1 , 2 ) = ∞ and lim N 1 →∞ N a ( N 1 ) = I a d ( ν a ,ν 1 ) .

We now consider the allocation where every arm a ∈ B has N a ( N 1 ) samples, and consider the function,

Observe that, for all a ∈ B , d ( ν 1 ,x 1 ,a ) d ( ν a ,x 1 ,a ) is strictly decreasing for N 1 ≥ N 1 , 2 . Also if B c is non-empty, then every term d ( ν 1 ,x 1 ,a ) d ( ν a ,x 1 ,a ) with N a > 0 is strictly decreasing in N 1 . As a result, the overall function N 1 → h ( N 1 ) is strictly decreasing.

Moreover, as N 1 increases to ∞ , N a ( N 1 ) converges to I a d ( ν a ,ν 1 ) for every a ∈ B . Hence, h ( N 1 ) decreases to 0 as N 1 →∞ . Therefore, if we can show that h (max { N 1 , 1 , N 2 , 1 } ) ≥ 1 , then we can find a unique N ⋆ 1 ≥ max { N 1 , 1 , N 1 , 2 } at which h ( N ⋆ 1 ) = 1 , and can take N ⋆ a = N a ( N ⋆ 1 ) for every a ∈ B . Following this, to prove uniqueness, it is sufficient to show h (max { N 1 , 1 , N 1 , 2 } ) ≥ 1 .

If N 1 , 2 ≥ N 1 , 1 , then some a ∈ B has N a ( N 1 , 2 ) = ∞ . As a result, we have h ( N 1 , 2 ) = ∞ ≥ 1 . Otherwise, if N 1 , 1 ≥ N 1 , 2 > 0 , then by definition of N 1 , 1 , we have

Hence we finish proving the first part of Theorem D.1.

To see the necessity of the stated optimality conditions for O observe that we cannot have N ⋆ 1 = 0 or N ⋆ a = 0 as that implies that the index W a ( N ⋆ 1 , N ⋆ a ) is zero. Further, if W a ( N ⋆ 1 , N ⋆ a ) > log(1 / (2 . 4 δ )) , the objective improves by reducing N ⋆ a . Thus the constraints (20) must be tight. To see the tightness of (20) again note that the derivative of W a ( N 1 , N a ) with respect to N 1 and N a , respectively, equals d ( µ 1 , x 1 ,a ) and d ( µ a , x 1 ,a ) .

Now, perturbing N 1 by a tiny ϵ and adjusting each N a by d ( µ 1 ,x 1 ,a ) d ( µ a ,x 1 ,a ) ϵ maintains the value of W a ( N ⋆ 1 , N ⋆ a ) . Thus, at optimal N ⋆ , necessity of tightness of inequalities in (20) follows. This condition can also be seen through the Lagrangian (see [2]).

The fact that these three criteria uniquely specify the optimal solution follows from our analysis above. Since the convex problem O has a solution, the uniqueness of the solution above satisfying the necessary conditions implies that this uniquely solves O .

To prove Propositions 2.1 and 2.2, we need to use the Implicit function theorem. For that, we define the following functions,

̸

where N = ( N a ∈ R ≥ 0 : a ∈ [ K ]) , I ∈ R , N ∈ R + , and, for every a ∈ [ K ] / { 1 } , x 1 ,a ( N 1 , N a ) = N 1 µ 1 + N a µ a N 1 + N a .

Using these functions, for every non-empty B ⊆ [ K ] / { 1 } , we define the vector valued function

We call J [ K ] / { 1 } ( · ) as J ( · ) . Recall that B denotes B ∪ { 1 } .

For every m ≥ 1 , we use the notation 0 m to denote a m -dimensional vector with all entries set to zero. Observe that, for every B ⊆ [ K ] / { 1 } , J B ( N , I, N ) = Φ B ( N , I, 0 K , N ) , where the function Φ B ( · ) is defined in Appendix C.

Lemma D.1 is essential for proving Proposition 2.1.

Lemma D.1. For every N > 0 and non-empty B ⊆ [ K ] / { 1 } , if ˆ N = ( ˆ N a ∈ R ≥ 0 : a ∈ [ K ]) satisfies ∑ a ∈ B c ˆ N a < N , and J B ( ˆ N , ˆ I, N ) = 0 | B | +1 , then, the Jacobian of J B ( · ) with respect to the arguments ( N B , I ) is invertible at ( ˆ N , ˆ I, N ) .

Proof. We have J B ( N , I, N ) = Φ B ( N , I, 0 K , N ) , for every tuple N , I, N , and non-empty B ⊆ [ K ] / { 1 } . As a result, Lemma D.1 follows from statement 3 of Lemma C.1.

For every non-empty subset B ⊆ [ K ] / { 1 } , we define the function ˜ J B ( · ) to be the first | B | + 1 components of the vector valued function J B ( · ) , or in other words,

Observe that ˜ J B ( · ) depends only on the tuple N and I , and doesn't depend on N . Also for every tuple ( N , I ) ∈ R K ≥ 0 , ˜ J B ( N , I ) = ˜ Φ B ( N , I, 0 K ) , where ˜ Φ B is defined in Appendix C.

We now proceed on proving Propositions 2.1 and 2.2.

Proof of Proposition 2.1: Observe that, for every non-empty B ⊆ [ K ] / { 1 } , solving the system (2) is equivalent to solving for the pair N B , I B in J B (( N B , N B c ) , I B , N ) = 0 .

For every I ≥ 0 , by Theorem D.1, there is a unique N B = ( N a ∈ R ≥ 0 : a ∈ B ) for which, ˜ J B (( N B , N B c ) , I ) = 0 | B | +1 . We denote that solution using N B ( I ) (we supress the dependence on N B c for cleaner presentation, and also because we will be treating N B c like a constant in the rest of the proof). Since, ˜ J B ( N , I ) = ˜ Φ B ( N , I, 0 K ) , by the Implicit function theorem and using statement 1 of Lemma C.1, the function I → N B ( I ) is continuously differentiable. Also, we have

where the right most quantity is evaluated at the tuple (( N B ( I ) , N B c ) , I, 0 K ) . Moreover, using statement 2 of Lemma C.1, we have,

As a result, the function ∑ a ∈ B N a ( I ) is strictly increasing in I with a derivative atleast ∑ a ∈ B 1 d ( µ a ,µ 1 ) . Also, for I = 0 , the unique solution is N 1 (0) = N 1 , 1 and N a (0) = 0 for every a ∈ B . As a result, as I increases from 0 to ∞ , ∑ a ∈ B N a ( I ) increases from N 11 to ∞ monotonically. Hence, for every N ≥ N 11 + ∑ a ∈ B c N a , we can find a unique I B for which ∑ a ∈ B N a ( I B ) + ∑ a ∈ B c N a = N . Therefore N B ( I B ) , I B becomes the unique tuple to satisfy, J B (( N B , N B c ) , I B , N ) = 0 | B | +2 .

Proof of Proposition 2.2: Recall that solving O in (20) is equivalent to solving (1). With this observation Proposition 2.2 follows directly from the definition of N [ K ] / { 1 } (1) , I [ K ] / { 1 } (1) and the second statement of Theorem D.1.

D.1 Single variable formulation of the lower bound problem and intuition behind the anchor function

Now we show that the K -variable convex optimization problem O defined in (20) can be reduced to a single variable convex optimization problem involving only N 1 . To see this, observe that O is equivalent to the problem

̸

This follows from the proof of Theorem D.1.

We first make some observations about O 1 .

Upon fixing some N 1 > 0 , N a → W a ( N 1 , N a ) is strictly increasing for every a = 1 . Moreover, we have

̸

̸

̸

As a result, every feasible solution of O 1 must satisfy N 1 d ( µ 1 , µ a ) > log(1 / (2 . 4 δ )) for every a = 1 , which implies N 1 > log(1 / (2 . 4 δ )) min a =1 d ( µ 1 ,µ a ) .

Further,

̸

Using the preceding observation, for every a = 1 and N 1 > log(1 / (2 . 4 δ )) min a =1 d ( µ 1 ,µ a ) , we can find a unique N a such that W a ( N 1 , N a ) = log(1 / (2 . 4 δ )) . We use ¯ N a ( N 1 ) to denote that unique N a as a function of N 1 . Using the Implicit function theorem, it is easy to prove that the function N 1 → ¯ N a ( N 1 ) is differentiable for every N 1 > log(1 / (2 . 4 δ )) min a =1 d ( µ 1 ,µ a ) with the derivative being

̸

It follows that O 1 is equivalent to the single variable optimization problem:

̸

̸

Plot of f(N1)

Lowerbound

O2's objective:f(N1)

No. of samples to first arm: N1

̸

̸

which is exactly negative of the anchor function defined in Section 2 w.r.t. the allocation where arm 1 gets N 1 samples and every sub-optimal arm a = 1 gets ¯ N a ( N 1 ) samples. Furthermore, f ′ ( N 1 ) is strictly increasing in N 1 and increases from -∞ to 1 as N 1 increases from log(1 / (2 . 4 δ )) min a =1 d ( µ 1 ,µ a ) to ∞ . As a result, O 2 is a convex problem w.r.t. N 1 and the optimal N 1 solving O 2 is uniquely identified by the condition:

̸

̸

which is equivalent to saying that the anchor function g must be zero when evaluated at the allocation where the first arm gets N 1 samples and every arm a = 1 gets ¯ N a ( N 1 ) samples. Figure 3 shows an illustrative plot of O 2 's objective f ( N 1 ) .

This observation also implies that the unique β which makes the β -EB-TCB(I) policy in [16] asymptotically optimal is uniquely identified by the first order conditions: 1) the anchor function g must be zero and 2) indexes of the sub-optimal arms must be equal. Hence allocations made by every

̸

̸

̸

asymptotically optimal sampling policy must converge to these first order conditions. Both the fluid dynamics in Section 4 and Proposition 5.1 in Section 5, respectively, show that the sampling policies of (I)AT2 algorithm converges to these first order conditions in a fluid model and in the proposed algorithms.

D.2 Sub-optimality of TCB(I)

[22] implicitly assumes that the optimal proportion is uniquely identified by the condition that indexes of every sub-optimal arm under the proportion are equal. Note that the optimal proportion is a K dimensional vector and the conditions mentioned in [22, Lemma 2, statement 2] has K -1 equations ( K -2 equations to maintain equality of K -1 indexes and one more equation to make sure that all the entries in the K -dimensional vector add up to 1 ). Moreover, in Appendix D.1, changing the variables of the problem O 2 to capture proportion of samples allocated to every arm, it is not hard to prove that for every value of w 1 ∈ (0 , 1) , we can get a unique proportion ( w 1 , w 2 , . . . , w K ) such that ∑ a ∈ [ K ] w a = 1 and index of all sub-optimal arms under the proportion are equal. By the argument in Appendix D.1, the unique optimal proportion out of these infinite no. of proportions satisfying equality of the indexes, is uniquely identified by the necessary and sufficient condition that the anchor function g ( · ) evaluated at that proportion must be zero. Without this condition, the allocation cannot be optimal. In the numerical experiments of Appendix J.2, we see that the anchor function doesn't always converge to zero for the TCB(I) algorithm. As a result, allocations made by TCB(I) can be sub-optimal.

E Proofs from Section 4

Proof of Theorem 4.1: We first prove all the steps of Theorem 4.1 except for showing the existence of β > 0 and independent of N such that I ′ b ( N ) > β . That requires intermediate lemmas and is done separately.

First suppose that B contains a singleton index b . Define N 1 ( N ) and N b ( N ) using IFT through the equations

̸

and ∑ a N a = N. For each a , letting x ′ 1 ,a denote the derivative of x 1 ,a with respect to N 1 , ˜ x ′ 1 ,b denote the derivative of x 1 ,b with respect to N b . It is easy to check that

and each x ′ 1 ,a = N a ∆ a ( N 1 + N a ) 2 . Differentiating (24) with respect to N , observing that N ′ b = 1 -N ′ 1 , we get

̸

̸

̸

as stipulated. Also N ′ b = ∑ a =1 h a N a ∑ a =1 h a N a + N 1 h b .

̸

Now consider the case where g = 0 , and we have set B ⊂ [ K ] / 1 of indices where the indexes are equal, they are higher for the remaining set. Cardinality of B is at least 2. We want to argue that as N increases, and the equality of indexes in B is maintained along with g = 0 , then the tied indexes will increase with N .

We have for b, a ∈ B

It follows that

Furthermore, g = 0 , i.e., with

Then,

Furthermore, let

Let x ( ˜ a ) be such that

̸

Keeping a particular b ∈ B fixed, differentiating with respect to N , (since for each a , by definition of x 1 ,a , N 1 d ′ ( µ 1 , x 1 ,a ) + N a d ′ ( µ a , x 1 ,a ) = 0 ) we see from (26) that

Using (26) again in the above equality,

Then from (27), we have that

̸

(Recall that for each a , x ′ 1 ,a denotes the derivative of x 1 ,a with respect to N 1 and ˜ x ′ 1 ,a denotes the derivative of x 1 ,a with respect to N a .)

Plugging (28) and (25) in (29), multiplying each term by N 2 1 , we see that N ′ 1 is a ratio of

In particular, since, ∑ a N ′ a = 1 , (3) follow. Statement 3 of Theorem 4.1 follows from (3) and expression for I a ( N ) . Since I ′ a ( N ) > 0 index is non-decreasing in N . Furthermore, lim N →∞ I a ( N ) = N a d ( µ a , µ 1 ) .

To prove the existence of β > 0 and independent of N such that I ′ b ( N ) > β , we need Lemmas (E.1), (E.2) and (E.3). In Lemma E.3, we argue that the indexes in set B grow linearly with the number of samples. Since index for arm a ∈ B c are bounded, eventually indexes in set B catch-up with other indexes.

Some notation first. Observe that d ( µ 1 , x ) -d ( µ a , x ) is a continuous and strictly decreasing function of x ∈ [ µ a , µ 1 ] . It equals d ( µ 1 , µ a ) for x = µ a and -d ( µ a , µ 1 ) for x = µ 1 . Let x a ∈ ( µ a , µ 1 ) be such that

It is guaranteed to exist since the ratio d ( µ a , x ) /d ( µ 1 , x ) ∈ (0 , ∞ ) is monotonic in x .

Next, let

Since K ≥ 2 , we have x ( ˜ a ) ≥ x ˜ a , and d ( ˜ a ) ≤ d ˜ a .

Lemma E.1. Suppose that g = 0 . Then,

̸

Lemma E.2. Suppose that g = 0 . Then,

there exist constants H and D such that h a ≤ H for all a , and d -1 a,a ≤ D for all a .

Further, there exists an ˜ a such that N ˜ a > αN for some α > 0 and the corresponding h ˜ a is bounded from below by a positive constant.

Lemma E.3. Suppose that g = 0 , and for B ⊂ [ K ] / 1 the indexes are all equal and are strictly higher for the remaining set. Then there exists a constant β > 0 such that

Proof of Lemma (E.1): Since g = 0 , it follows that for each a ∈ [ K ] \ 1 ,

Thus, x 1 ,a ≥ x a . This in turn implies that for each a ,

̸

The above follows from substituting for x 1 ,a in the inequality x 1 ,a ≥ x a , and from the definition of d a . Moreover, since g = 0 , it also follows that for each a = 1 , x (( ˜ a ) ≥ x 1 ,a , implying

̸

Then, and

and the result follows.

Proof of Lemma (E.2): Recall the definitions of h a and f ( µ , a, N ) from Section 4.

Since, g = 0 implies that x 1 ,a ≥ x a for all a , it follows that d a,a = d ( µ a , x 1 ,a ) ≥ d ( µ a , x a ) . In particular, for all a

Further, d ′ ( µ a , x 1 ,a ) is continuous in x 1 ,a and is bounded from above by sup x a ≤ x 1 ,a ≤ µ 1 d ′ ( µ a , x 1 ,a ) . Similarly, -d ′ ( µ 1 , x 1 ,a ) is bounded from above by sup x a ≤ x 1 ,a ≤ µ 1 -d ′ ( µ 1 , x 1 ,a ) . This implies that f ( µ , a, N ) is bounded from above by a positive constant and hence so is h a .

To see part 2, observe from definition of x (˜ a ) that x 1 , ˜ a ≤ x (˜ a ) . It follows that

N ˜ a ≥ N 1 d -1 ˜ a . Therefore,

̸

Again, x 1 , ˜ a ≤ x ( ˜ a ) . Therefore, d ˜ a, ˜ a = d ( µ ˜ a , x 1 , ˜ a ) ≤ d ( µ ˜ a , x ( ˜ a )) and d 1 , ˜ a = d ( µ 1 , x 1 , ˜ a ) ≥ d ( µ 1 , x ( ˜ a )) .

Further, d ′ ( µ ˜ a , x 1 , ˜ a ) is continuous in x 1 , ˜ a and is bounded from below by

Similarly, -d ′ ( µ 1 , x 1 , ˜ a ) is bounded from below by inf x ˜ a ≤ x 1 , ˜ a ≤ x ( ˜ a ) -d ′ ( µ 1 , x 1 , ˜ a ) .

Thus, f ( µ , ˜ a ) is bounded from below. Further, since each N a ≤ d a N 1 , N 2 1 / ( N 1 + N ˜ a ) 2 ≥ (1 + d ˜ a ) -2 , hence, h ˜ a is bounded from below by a positive constant.

Proof of Lemma (E.3): Recall from (31) that

̸

Also, recall the definition of f ( µ , a, N ) and h a from Section 4.

Because of (30) and (32), and since N ˜ a ≤ N , we see that f ( µ , ˜ a ) is bounded from below by a positive constant, and the same is true for h ˜ a .

If ˜ a ∈ B , recall that h B = ∑ a ∈ B h a d -1 a,a . Thus, h B is greater than a constant times N . This ensures that N ′ ˜ a is bounded from below by a positive constant. Since d ˜ a, ˜ a is also bounded from below by a positive constant, we conclude that there exists a β > 0 such that I ′ B ( N ) > β .

If ˜ a / ∈ B , then recalling that h ( N ) = ∑ a ∈ B c / 1 h a N a , we conclude that N ′ ˜ a is bounded from below by a positive constant. This implies that as N increases by a positive fraction, so does each N a for a ∈ B . This in turn ensures that then h B is thereafter bounded from below by a positive constant. In particular, after some delay we have I ′ B ( N ) > β for some β > 0 .

E.1 Fluid dynamics starting at g < 0

Proposition E.1 provides us the ODEs by which the fluid allocations evolve in step 3 of the description of fluid dynamics when g < 0 .

Proposition E.1. Now consider the case where g < 0 at total allocations N , and B again denotes the set of arms whose indexes have the minimum value. Then, till I B ( N ) increases with N to either hit an index in B c , or for g to equal zero, whichever is earlier, I ′ B ( N ) = (∑ a ∈ B d -1 a,a ) -1 , and for a ∈ B , N ′ a = d -1 a,a (∑ a ∈ B d -1 a,a ) -1 . In particular, I B ( N ) and each ( N a , a ∈ B ) , are increasing functions of N .

Proof. Let i 1 denote the arm corresponding to a minimum index. Recall that ω ⋆ = ( ω ⋆ a : a ∈ [ K ]) denote the optimal proportions to the lower bound problem. Consider ˆ N a = ω ⋆ a ω ⋆ 1 N 1 . Recall that at these samples, g = 0 and all the indexes are equal. Let ˆ I denote the corresponding value of the indexes at this allocation.

First we argue that N i 1 < ˆ N i 1 .

Suppose this is not true, then g < 0 implies that for N 1 fixed, there exists an arm a so that N a < ˆ N a , else if each N a ≥ ˆ N a then since g increases with N a , we would have g ≥ 0 . This contradiction implies that index for arm a is < ˆ I B ( N ) . It follows that the index corresponding to i 1 is < ˆ I B ( N ) . Since the index increases with N i 1 , it follows that N i 1 < ˆ N i 1 .

Thus, initially N increases due to increase in N i 1 . Let B = { i 1 } . Suppose, iteratively that B = { i 1 , . . . , i j -1 } , denotes the smallest indexes that are equal and increase with N and g < 0 . Proof follows by observing that the derivative of each index a ∈ B satisfies the relation N ′ a d a,a = I ′ B ( N ) . Further, ∑ a ∈ B N ′ a = 1 .

Thus, as N increases, each N a ( N ) , a ∈ B increases, so that g increases. Since all indexes corresponding to B c are constant, as N increases, either g = 0 first or another index I j becomes equal to I B ( N ) .

E.2 Fluid dynamics of the β -EB-TCB algorithm ([16])

For every β ∈ (0 , 1) and allocation N = ( N a ∈ R ≥ 0 : a ∈ [ K ]) , we define the β -anchor function as,

Note that, if g ( N ; β ) = 0 , then β -fraction of the total no. of samples in the allocation N is allocated to the first arm. The fluid dynamics for the β -EB-TCB algorithm (see [16]) can be constructed similarly to that of the Anchored Top Two algorithm, by replacing the anchor function g ( µ , N ) with the β -anchor function g ( N ; β ) in Section 4.

Existence of fluid dynamics: Recall that, for every B ⊆ [ K ] / { 1 } , B denotes the set B ∪ { 1 } . Lemma E.4 and Proposition E.2 are essential for constructing the fluid behavior for β -EB-TCB algorithm.

Lemma E.4. Given a non-empty B ⊆ [ K ] / { 1 } , some tuple N B c = ( N a ∈ R ≥ 0 : a ∈ B c ) and I ≥ 0 , there is a unique tuple N B = ( N a ∈ R ≥ 0 : a ∈ B ) which satisfies,

where x 1 ,a = N 1 µ 1 + N a µ a N 1 + N a .

Moreover, if we define N 1 , 1 = β ∑ a ∈ B c N a and N 1 , 2 = I min a ∈ B d ( µ 1 ,µ a ) , then N 1 ≥ max { N 1 , 1 , N 1 , 2 } .

Proof. Proof of this lemma follows an argument similar to the proof of Theorem D.1.

First we fix some I ≥ 0 and N 1 ≥ N 1 , 2 . Note that for every a ∈ B , as N a increases from 0 to ∞ , N 1 d ( µ 1 , x 1 ,a ) + N a d ( µ a , x 1 ,a ) increases monotonically from 0 to N 1 d ( µ 1 , µ a ) . Since N 1 ≥ N 1 , 2 ≥ I d ( µ 1 ,µ a ) , we have N 1 d ( µ 1 , µ a ) ≥ I . This implies, for a given N 1 , there is a unique value of N a for which N 1 d ( µ 1 , x 1 ,a ) + N a d ( µ a , x 1 ,a ) = I . We call this value N a ( N 1 ) for every a ∈ B .

Observe that N 1 → N a ( N 1 ) is a strictly decreasing function of N 1 , and if N 1 = N 1 , 2 , then there exists an a ∈ B for which N a ( N 1 , 2 ) = ∞ . On the other hand, if N 1 →∞ , N a ( N 1 ) → I d ( µ a ,µ 1 ) for every a ∈ B .

For every N 1 , we consider the function

Note that h ( N 1 ; β ) is the value of g ( N ; β ) , when the tuple N has N a = N a ( N 1 ) for every a ∈ B . Note that N 1 → h ( N 1 ; β ) is strictly decreasing for N 1 ≥ N 1 , 2 . Moreover, as N 1 →∞ , h ( N 1 ; β ) → β -1 < 0 . In the rest of the argument, we show that h (max { N 1 , 1 , N 1 , 2 } ; β ) ≥ 0 . After we prove this, we can find a unique N 1 for which h ( N 1 ; β ) = 0 . Using this, we take N a = N a ( N 1 ) for every a ∈ B to obtain our unique tuple N B .

Now we consider two cases.

Case 1: If N 1 , 1 ≥ N 1 , 2 , then at N 1 = N 1 , 1 ,

As a result,

which implies h ( N 1 , 1 ; β ) ≥ 0 .

Case 2: If N 1 , 2 ≥ N 1 , 1 , then, as we argued before, there exists an a ∈ B for which N a ( N 1 , 2 ) = ∞ . As a result,

implying h ( N 1 , 2 ; β ) = β > 0 .

Proposition E.2 stated below is crucial for constructing the fluid dynamics of the β -EB-TCB policy and is analogous to Proposition 2.1 used for constructing the fluid dynamics of the anchored top-two algorithm.

Proposition E.2. For every non-empty B ⊆ [ K ] / { 1 } , tuple N B c = ( N a ∈ R ≥ 0 : a ∈ B c ) , and N ≥ 1 1 -β ∑ a ∈ B c N a , there exists a unique tuple N B = ( N a ∈ R ≥ 0 : a ∈ B ) and I B ≥ 0 for which,

If we denote that tuple by N B ( N ) and I B ( N ) , then the functions N → N B ( N ) , I B ( N ) are continuously differentiable with respect to N .

Proof. Proof of Proposition E.2 follows by an argument similar to the one used in the proof of Proposition 2.1, by using Lemma E.4 instead of Theorem D.1. Observe that the β -anchor function g ( N ; β ) is strictly decreasing in N 1 and strictly increasing in N a , when N 1 > 0 . As a result, statement 1 of Lemma C.1 in Appendix C holds true upon having,

and defining the set Z B using the modified function Φ B .

With the above modification, if we can find a constant γ > 0 such that, 1 T | B | +1 ( ∂ ˜ Φ B ∂ N B ) -1 ∂ ˜ Φ B ∂I ≤ -γ < 0 for every tuple in Z B , then statements 2 and 3 of Lemma C.1 also hold true for this modified Φ B . As a result, Proposition E.2 follows using Lemma E.4 by the same argument used for proving Proposition 2.1 using Theorem D.1.

In the rest of the proof we argue the existence of such a constant γ > 0 .

Let v = ( v a : a ∈ B ) ∈ R | B | +1 be the solution to the system

We have ∂Φ 1 ∂ N B v = ∂Φ 1 ∂I , and N 1 = β ∑ a ∈ [ K ] N a , which after some algebraic manipulation implies,

Therefore proving that v 1 is upper bounded by a negative constant is sufficient for proving the desired result.

(33) further implies,

For every a ∈ B , we have which implies

where x 1 ,a = N 1 µ 1 + N a µ a N 1 + N a . We use d 1 ,a and d a,a , respectively, to denote d ( µ 1 , x 1 ,a ) and d ( µ a , x 1 ,a ) . For every a ∈ B , we can eliminate v a for (33) using (34). After this procedure, we get,

̸

̸

Since N 1 = β ∑ a ∈ [ K ] N a , using (6) we have d a,a = Θ (1) for all a ∈ [ K ] / { 1 } , where the constant hidden in Θ ( · ) is independent of N . As a result, by (35), -v 1 = Ω (1) . Hence we conclude the proof.

Constructing the fluid ODEs: Without loss of generality, we assume that the fluid dynamics starts from a state N where g ( N ; β ) = 0 . Otherwise,

If g ( N ; β ) > 0 , the algorithm gives samples to arm 1 till g ( N ; β ) = 0 .

If g ( N ; β ) < 0 , the β -EB-TCB algorithm follows the dynamics in Proposition E.1, and reaches g ( N ; β ) = 0 in a finite amount of time.

Following Proposition E.2, the algorithm tracks the allocation N B ( N ) at a given time N , where B denotes the set of minimum index arms. We now determine the ODEs by which the state of the algorithm evolves.

To simplify the notations, we use g β as a shorthand for g ( N ( N ); β ) at a given time N . For every a ∈ [ K ] , we use N a , N ′ a , I B , and I ′ B , respectively, to denote N a ( N ) , N ′ a ( N ) , I B ( N ) and I ′ B ( N ) . For every a ∈ B c , we use I a to denote I a ( N ) . Also for every a ∈ [ K ] / { 1 } , we adopt the notation d 1 ,a and d a,a , respectively, for the quantities d ( µ 1 , x 1 ,a ) and d ( µ a , x 1 ,a ) , where x 1 ,a = N 1 µ 1 + N a µ a N 1 + N a .

For every non-empty B ⊆ [ K ] / { 1 } we define the quantity,

We now show the fluid ODEs in the following proposition.

Proposition E.3 ( Fluid ODEs for β -EB-TCB ) . Let us assume the algorithm starts from a state N ( N 0 ) = ( N 0 a : a ∈ [ K ]) with ∑ a ∈ [ K ] N 0 a = N 0 , N 0 1 = βN 0 and N 0 > 0 . Let B ⊆ [ K ] / { 1 } be the set of arms having minimum index at a given time N ≥ N 0 . The following statements hold true about the allocation N B ( N ) = ( N a ( N ) : a ∈ B ) made by β -EB-TCB algorithm,

The allocation N ( N ) = ( N a ( N ) : a ∈ [ K ])

The index I B ( N ) of the arms in B evolves by the following ODE,

There exists a constant c > 0 such that, I ′ B ≥ c . On the other hand, indexes of the arms in B c remains upper bounded by N 0 a d ( µ a , µ 1 ) .

Proof. Statement 1: N ′ = β follows directly from the fact that g = 0 .

By definition of B

1 β , we have

for every a, b ∈ [ K ] . Taking derivative on both sides, we get,

Using the above expression in (39), we get,

Since N ′ 1 = β and N 1 = βN (which follows from g β = 0 ), the above equation implies,

Adding both sides for a ∈ B , we get,

which implies

Statement 2: We know

Putting in the derivatives from (36), we obtain,

which implies,

Using (38), we have

Statement 3: For the following argument, the constants hidden in O ( · ) , Ω ( · ) and Ω ( · ) are independent of N .

Note that N 1 = βN . As a result, using (6), d a,a = Θ (1) for every a ∈ [ K ] / { 1 } . This implies, we can find a constant c 1 > 0 such that d B ≥ c 1 . On the other hand, using (12), we have,

for every a ∈ B . Since N 1 = βN and N a ≤ N , we have

Adding for all a ∈ B , we get I B = Θ ( ∑ a ∈ B N a ) . Therefore, I B ≥ c 2 ∑ a ∈ B N a , for some constant c 2 > 0 .

Now using (37),

Taking c = min { c 1 , c 2 } > 0 , we have the desired result.

Now for arms a ∈ B c , note that x 1 ,a = arg min x ∈ [ µ a ,µ 1 ] ( N 1 d ( µ 1 , x ) + N 0 a d ( µ a , x ) ) and I a = N 1 d ( µ 1 , x 1 ,a ) + N 0 a d ( µ a , x 1 ,a ) .

As a result, putting x = µ 1 , we have I a ≤ N 0 a d ( µ a , µ 1 ) .

Reaching β -optimal proportions: By statement 3 of Proposition E.3, the indexes of the arms in B increase at a linear rate, whereas the indexes of the arms in B c stay bounded above by a constant. As a result, by some finite time, I B crosses the index of some arm in a ∈ B c , after which B gets updated to B ∪ { a } . The same process then continues with the updated B . In this way, eventually B = [ K ] / { 1 } and the fluid dynamics reaches the β -optimal proportion ω ⋆ ( β ) = ( ω ⋆ a ( β ) : a ∈ [ K ]) ( β -optimal proportion is the solution to the max-min problem (1) with the added constraint ω 1 = β ) where,

Applying the same argument as used to bound the time to reach optimal proportions in Section 4, if the fluid dynamics start at some time N 0 with state N ( N 0 ) = ( N a ( N 0 ) : a ∈ [ K ]) , then it reaches stability by a time atmost N 0 min a ∈ [ K ] ω ⋆ a ( β ) .

F Intuitions based on fluid dynamics applied to algorithmic behavior

F.1 Indexes once they meet do not separate

In the fluid dynamics described in Theorem 4.1, once the indexes meet thereafter they move up together by construction. It turns out that I ′ B ( N ) is positive. Below we give a heuristic argument that in our fluid dynamics, once a set of smallest indexes that are equal, increase and catch up with another index, their union then remains equal and increases together with N . This argument provides important insights which help us later to prove that after after a random time of finite expectation, if our algorithm picks a suboptimal arm, then it picks that arm again in a periodic manner, which helps us prove closeness of indexes w.r.t. the algorithmic allocations (see Lemma 5.1). Differentiating g = 0 with respect to N , we see that,

̸

̸

By our previous discussion

̸

Inductively, suppose that a set B of indexes are moving up together and they run into another index b at time N . Upon assuming contradiction, we can have a neighbourhood [ N,N + ∆N ] where the algorithm only allocates to a subset C ⊂ B ∪{ b } and doesn't allocate to arms in D = B ∪{ b }-C = ∅ . Then the allocations follow the ODEs in (3) of Theorem 4.1 with B = C , in the interval [ N,N + ∆N ] .

Consider the probability vector ( p a : a ∈ [ K ] / { 1 } ) where,

̸

Note that p a > 0 for every a ∈ [ K ] / { 1 } . We have from (40) that

Letting b = arg max a ∈ C N ′ a ( N ) N a ( N ) (where N ′ a ( N ) is the derivative in (3) of Theorem 4.1, upon putting B = C ), we have

̸

where the strict inequality in (1) follows from the fact that D = B ∪ { b } -C = ∅ , causing ∑ a ∈ C p a < 1 .

̸

We now argue that D must be empty. Suppose instead that D = ∅ and a ∈ D . Because all indexes in B are equal at time N , we have, N 1 d ( µ 1 , x 1 ,a ) + N a d ( µ a , x 1 ,a ) = N 1 d ( µ 1 , x 1 ,b ) + N b d ( µ b , x 1 ,b ) at N. Observe that for any arm d ∈ [ K ] / { 1 } , derivative of its index with respect to N equals N ′ 1 d ( µ 1 , x 1 ,d ) + N ′ d d ( µ d , x 1 ,d ) (since, by definition of x 1 ,d , N 1 d 2 ( µ 1 , x 1 ,d ) + N d d 2 ( µ d , x 1 ,d ) = 0 ). Since arm a gets no sample in [ N,N + ∆N ] , we have N ′ a = 0 , which implies

We now argue that N ′ 1 d ( µ 1 , x 1 ,a ) is strictly less than (43) at N . As a result, if ∆N > 0 is picked sufficiently small, index of b , which is the minimum index, outruns index of a in [ N,N + ∆N ] .

Consider the difference

We want show that this is strictly negative. We consider two cases,

Case I: If d ( µ 1 , x 1 ,a ) -d ( µ 1 , x 1 ,b ) ≤ 0 , it follows trivially.

Case II: If d ( µ 1 , x 1 ,a ) -d ( µ 1 , x 1 ,b ) > 0 , since N ′ b > 0 , using (42) we can upper bound (44) by

Since the two indexes are equal at this point, we have

Substituting this in (45), the latter equals

We thus have our contradiction. Therefore, indexes of the arms in B ∪ { b } move together.

F.2 Proof sketch of Lemma 5.1

We consider the situation where the algorithm picks some arm a ∈ [ K ] / { 1 } at iteration N and doesn't pick a for the next R ( N ) ≥ 1 iterations. For better readability, we denote R ( N ) using R . In the following argument, we try to bound R from above using N . We can prove that ˜ N j ( N ) = Θ ( N ) for every j ∈ [ K ] and N ≥ T good (see Remark G.1 in Appendix G.2). As a result, R is atmost O ( N ) for N ≥ T good . Below, we improve the upper bound to O ( N 1 -3 α/ 8 ) by a refined analysis.

Let us define ∆ ˜ N j ( N,t ) = ˜ N j ( N + t ) -˜ N j ( N ) for every j ∈ [ K ] and N,t ≥ 1 . By our choice of T good , we have | g ( µ , ˜ N ( N )) | = O ( N -3 α/ 8 ) for N ≥ T good (see Remark G.1 in Appendix G.2). By applying mean value theorem on g ( · ) for N ≥ T good , we have,

̸

̸

where (ˆ p j ( N,t ) : j ∈ [ K ] / { 1 } ) is a probability distribution over the set [ K ] / { 1 } , depending on N and t (this distribution is not important to the discussion and is spelt out at Appendix G.6). Taking b t = arg max a =1 ∆ ˜ N a ( N,t ) ˜ N a ( N ) , and using (46), we obtain,

Observe that (46) and (47), respectively, resembles (41) and (42) from Appendix F.1, except for a O ( N 1 -3 α/ 8 ) term due to the noise in ˜ µ .

Applying the mean value theorem, we can bound the difference between the empirical indexes of arm a and b t at iteration N + t by,

Now if d ( ˜ µ 1 , ˜ x 1 ,a ) -d ( ˜ µ 1 , ˜ x 1 ,b t ) < 0 , (48) implies,

Otherwise, if d ( ˜ µ 1 , ˜ x 1 ,a ) -d ( ˜ µ 1 , ˜ x 1 ,b t ) ≥ 0 , using (47) we have

Note that (48) resembles (45) in Appendix F.1. Since I a ( N -1) ≤ I b t ( N -1) , we can prove using the mean value theorem that I a ( N ) ≤ I b t ( N ) + O ( N 1 -3 α/ 8 ) . Now expanding the empirical indexes, we get

Now dividing both sides by ˜ N b t ( N ) and using the fact that ˜ N b t ( N ) = Θ ( N ) , we have

Using the above inequality in (49), we have,

whenever ˜ 1 ˜ 1 ,a ˜ 1 ˜ 1 ,b t

Since ˜ N j ( N ) = Θ ( N ) and ˜ µ j ( N ) ≈ µ j for all j ∈ [ K ] and N ≥ T good , the coefficient of ∆ ˜ N b t ( N,t ) in (49) and (50) are -Θ (1) . As a result, we can find a constant C 3 > 0 , such that, for t ≤ R and N ≥ T good ,

Applying the mean value theorem and using the fact that t can be atmost O ( N ) , we can prove that, (51) implies,

As a result, we have a constant C > 0 such that,

Using (47), we can choose T good suitably, and find constants D 1 , D 2 > 0 , such that, whenever N ≥ T good ,

We consider the case where R ≥ max { D 1 , 2 C 4 C 3 D 2 } × N 1 -3 α/ 8 .

Since R ≥ D 1 N 1 -3 α/ 8 , we have

For notational simplicity, we use b to denote b R . We consider the iteration N + S where arm b was selected for the last time before iteration N + R . Then by definition of b R and S , and using the above inequality, we have

̸

Also, since ∆ ˜ N b ( N,S ) = ∆ ˜ N b ( N,R ) and for every j = 1 ∆ ˜ N j ( N,R ) ≥ ∆ ˜ N j ( N,S ) , we conclude b = b S . Therefore,

The above inequality implies, the AT2 algorithm pulls arm b at iteration N + S , even though

which is contradicting the algorithm's description. Hence we must have

G Algorithmic allocations: non fluid behaviour

In the following sections, unless otherwise stated, the proof of the mentioned results for AT2 (1) and IAT2 (2) algorithms follow a similar argument. Also, the constants introduced while stating the results in the following sections might be different for the two algorithms.

While using the O ( · ) , Θ ( · ) and Ω ( · ) notations, we imply that the hidden constants can depend on the choice of algorithm among AT2 and IAT2, instance µ , exploration factor α ∈ (0 , 1) and no. of arms K , and are independent of the sample path.

G.1 Convergence of algorithmic allocations to the optimality conditions

In this section, our agenda is to prove the convergence of the allocations of AT2 and IAT2 algorithms to the optimality conditions mentioned in Proposition 2.2. For ease of presentation we state the conditions uniquely characterizing the optimal proportion ω ⋆ below according to Proposition 2.2:

̸

Recall that for every a ∈ [ K ] and iteration N , ˜ ω a ( N ) = ˜ N a ( N ) N denotes the proportion of samples allocated by the algorithm to arm a . Let ˜ ω ( N ) = ( ˜ ω a ( N ) : a ∈ [ K ]) .

Recall the anchor function g ( µ , ˜ N ( · )) and index I a ( · ) for every alternative arm a ∈ [ K ] / { 1 } . In Section 5, we defined the normalized index H a ( · ) of every arm a ∈ [ K ] / { 1 } at iteration N as H a ( N ) = 1 N I a ( N ) . In the next two sections, we prove,

and

a.s. in P µ as N → ∞ . Moreover, we show that, after a random time of finite expectation, both the convergences in (55) and (56) happen at a uniform rate over all sample paths. We prove these convergence results in Proposition G.1 and G.2 stated below.

Proposition G.1 ( Convergence of g to zero ) . There exists constants M 4 ≥ 1 and C > 0 independent of the sample paths, such that, if T 6 is defined as the iteration at which g ( ˜ µ ( · ) , ˜ N ( · )) crosses the value zero after iteration max { M 4 , T 5 } ( T 5 is a random time satisfying E µ [ T 5 ] < ∞ and defined in Definition G.1 of Appendix G.1.1), then for N ≥ T 6 we have,

Moreover, the random time T 6 satisfies E µ [ T 6 ] < ∞ .

Proposition G.2 ( Closeness of the indexes ) . There exists a random time T 8 (defined in Definition G.4 of Appendix G.1.2) satisfying E µ [ T 8 ] < ∞ , such that, for N ≥ T 8 , every pair of alternative arms a, b ∈ [ K ] / { 1 } has,

Proof of Proposition 5.1: By the definition of T stable in Definition G.5 of Appendix G.2, we have T stable ≥ T 6 , T 8 , where T 6 and T 8 are the random times mentioned, respectively, in Proposition G.1

̸

Proof of Proposition G.1 is in Appendix G.1.1. We prove a detailed version of Proposition G.2 as Proposition G.3 in Appendix G.1.2. Both these results are crucial later for proving the convergence of the algorithmic proportions ˜ ω ( N ) = ( ˜ ω a ( N ) : a ∈ [ K ]) to the optimal proportions ω ⋆ = ( ω ⋆ a : a ∈ [ K ]) in Proposition 3.1 from Section 3. We prove a detailed version of Proposition 3.1 as Proposition G.4 in Appendix G.2.

To prove Proposition G.1, G.2, and later Proposition G.4, we need to prove several technical properties related to exploration and the allocations made by the algorithms. The detailed technical results related to exploration are in Appendix G.4 and those related to the algorithmic allocations are in Appendix G.5. The arguments in Appendix G.1.1, G.1.2, and G.2 are self-contained, and we refer the reader to the related technical results whenever necessary. For ease of exposition, we provide below a brief summary of the statements proven in Appendix G.4 and G.5.

Summary of technical results in Appendix G.4 and G.5

We summarize below the results proven in Appendix G.4 and G.5 as events happening between the non-decreasing sequence of random times T 0 , T 1 , T 2 , T 3 , and T 4 , which are defined in Appendix G.5.

̸

T 0 def. = min { N ′ ≥ 1 | ∀ N ≥ N ′ , max a ∈ [ K ] | ˜ µ a ( N ) -µ a | ≤ ϵ ( µ ) N -3 α/ 8 } , where ϵ ( µ ) > 0 (defined in Appendix B), is a constant depending on the instance µ . By definition, we have ϵ ( µ ) ≤ 1 4 min a =1 ( µ 1 -µ a ) . As a result, the first arm becomes the empirically best arm and stays that way forever after iteration T 0 . In Lemma G.7 of Appendix G.3, we prove that E µ [ T 0 ] < ∞ , which implies T 0 < ∞ a.s. in P µ .

T 1 def. = max { T explo , T 0 } , where T explo < ∞ is a constant defined in Definition G.7 of Appendix G.4. After iteration T explo, the algorithm consecutively does exploration over a strech of atmost K iterations. Moreover, over a single such ' epoch ' of consecutive explorations, the algorithm explores every arm atmost once (follows from statement 1 and 3 of Proposition G.5). Note that E µ [ T 1 ] ≤ T explo + E µ [ T 0 ] < ∞ .

T 2 is defined in Lemma G.11 as the iteration at which the anchor function g ( ˜ µ ( · ) , ˜ N ( · )) crosses the value zero after the iteration max { M 1 , T 1 } ( M 1 ≥ 1 is a constant independent of the sample paths and defined in the proof of Lemma G.11). By Lemma G.9, there exists a constant C 1 ≥ 1 independent of the sample paths, such that T 2 ≤ C 1 max { M 1 , T 1 } . As a result, E µ [ T 2 ] ≤ C 1 ( M 1 + E µ [ T 1 ]) < ∞ . After iteration T 2 , the empirical anchor function g ( ˜ µ ( · ) , ˜ N ( · )) remains bounded inside an interval of the form [ -(1 -d min ) , d max -1] , where d min ∈ (0 , 1) and d max ∈ (1 , ∞ ) are constants independent of the sample paths (see Lemma G.11). Exploiting this, we argue that both ˜ N 1 ( N ) and max a ∈ [ K ] / { 1 } ˜ N a ( N ) become Ω ( N ) after iteration T 2 (see Corollary G.1).

T 3 def. = max { M 2 , T 2 } + 2 , where M 2 ≥ 1 is a constant chosen in the proof of Lemma G.12 and is independent of the sample paths. After iteration T 3 , whenever the algorithm picks an alternative arm a ∈ [ K ] / { 1 } , then for every other alternative arm b ∈ [ K ] / { 1 , a } , we have ˜ N b ( N ) ≥ γ ˜ N a ( N ) , for some constant γ ∈ (0 , 1) independent of the sample paths (see Lemma G.12). Note that E µ [ T 3 ] ≤ M 2 +2+ E µ [ T 2 ] < ∞ .

T 4 = C 2 ( T 3 +1) for some constant C 2 ≥ 1 independent of the sample paths, defined in Lemma G.13. After iteration T 4 , all the arms a ∈ [ K ] have ˜ N a ( N ) = Θ ( N ) (see Lemma G.13). Note that E µ [ T 4 ] ≤ C 2 ( E µ [ T 3 ] + 1) < ∞ .

G.1.1 Convergence of the anchor function to zero

The following lemma bounds the fluctuation of g ( ˜ µ , ˜ N ) around g ( µ , ˜ N ) due to the noise in the estimate ˜ µ of µ . We need this lemma later for proving convergence of the anchor function g to zero in Proposition G.1.

Lemma G.1 ( Bounding the noise in g ) . For every N ≥ T 2 (where T 2 is the random time defined in Lemma G.11 and satisfies E µ [ T 2 ] < ∞ ), we have,

Proof. Using mean value theorem for function of several variables, we have,

where ˆ µ a lies between µ a and µ a ( N ) for every a ∈ [ K ] .

˜ We define

Note that,

̸

and recall that d 1 ( · , · ) and d 2 ( · , · ) , respectively, denote the partial derivatives of d ( · , · ) with respect to its first and second argument.

By (6), for N > T 2 , we have,

By Corollary G.1 from Appendix G.5, we have ˜ N 1 ( N ) = Ω ( N ) for N > T 2 . As a result, d (ˆ µ a , ˆ x 1 ,a ) = Θ (1) for N > T 2 .

Moreover, for N > T 2 , we have: | d 1 (ˆ µ 1 , ˆ x 1 ,a ) | = O (1) , | d 1 (ˆ µ a , ˆ x 1 ,a ) | = O (1) (using (7)) ; | d 2 (ˆ µ 1 , ˆ x 1 ,a ) | = O (1) , | d 2 (ˆ µ a , ˆ x 1 ,a ) | = O (1) (using (8)) ; and d (ˆ µ 1 , ˆ x 1 ,a ) = O (1) (using (6)). As a result, for N > T 2 , all the partial derivatives in (57) are O (1) . Therefore, for N > T 2 ,

and hence completing the proof.

Halting of exploration: By Lemma G.13, for N ≥ T 4 , every arm a ∈ [ K ] has ˜ N a ( N ) = Θ ( N ) . As a result, we can find a constant λ ∈ (0 , 1) such that ˜ N a ( N ) ≥ λN for every a ∈ [ K ] and N ≥ T 4 . We choose M 3 large enough such that, for every N ≥ M 3 , λ ( N -1) > N α . Then we have min a ∈ [ K ] / { 1 } ˜ N a ( N -1) > N α for every N ≥ max { M 3 , T 4 +1 } . As a result, the algorithm doesn't do any exploration after iteration max { M 3 , T 4 + 1 } . With this, we define the following random time,

Definition G.1. We define T 5 = max { M 3 , T 4 +1 } .

Note that E µ [ T 5 ] < ∞ , since, E µ [ T 4 ] < ∞ .

We restate Proposition G.1 below,

̸

Statement of Proposition G.1. There exists constants M 4 ≥ 1 and C > 0 independent of the sample paths, such that, if T 6 denotes the iteration at which g ( ˜ µ ( · ) , ˜ N ( · )) crosses the value zero after iteration max { M 4 , T 5 } , then for N ≥ T 6 we have,

Moreover, the random time T 6 satisfies E µ [ T 6 ] < ∞ .

Proof. We prove the proposition via an inductive argument consisting of two main steps,

Initialization: We start with a choice of the constants C > 0 and M 4 ≥ 1 and show that g ( µ , ˜ N ( · )) satisfies (59) at iteration T 6 .

Induction: We show that, for every N ≥ T 6 , ∣ ∣ ∣ g ( µ , ˜ N ( N )) ∣ ∣ ∣ ≤ CN -3 α/ 8 implies ∣ ∣ ∣ g ( µ , ˜ N ( N +1)) ∣ ∣ ∣ ≤ C ( N +1) -3 α/ 8 .

By Lemma G.1, we have a constant C 1 > 0 independent of the sample path, such that,

By Lemma G.13, we have ˜ N a ( N ) = Θ ( N ) for every a ∈ [ K ] and N ≥ T 5 . As a result, by (15), we have constants C 2 , C ′ 2 > 0 independent of the sample paths, such that: for all N ≥ T 5 , and ˆ N a ∈ [ ˜ N a ( N -1) , ˜ N a ( N ) ] ,

where ˆ N = ( ˆ N a : a ∈ [ K ]) .

We use the constants C 1 , C 2 , and C ′ 2 as defined above in the rest of our proof.

Initialization: We choose C = 4 C 1 + C ′ 2 and M 4 = max { M 41 , M 42 , M 43 , M 44 } , where M 41 , M 42 , M 43 , M 44 are defined as,

M 41 ≥ 1 is the smallest number such that, for every N ≥ M 41 we have 2 C 1 N -3 α/ 8 > C 1 ( N -1) -3 α/ 8 ,

M 42 ≥ 1 is the smallest number such that, for every N ≥ M 42 we have C ( N +1) -3 α/ 8 ≥ ( C 1 + C ′ 2 ) N -3 α/ 8 ,

M 43 ≥ 1 is the smallest number such that, for every N ≥ M 43 we have C ( N +1) -3 α/ 8 ≥ C ′ 2 N -1 , and

M 44 ≥ 1 is the smallest number such that, for every N ≥ M 44 we have 3 Cα 8 ( N + 1) -(1+ 3 α 8 ) < C 2 N -1 .

By definition of T 6 , g ( ˜ µ ( · ) , ˜ N ( · )) has opposite signs at iterations T 6 -1 and T 6 . Therefore,

̸

Let a ∈ [ K ] be the arm pulled at iteration T 6 . Applying the mean value theorem we can find ˆ N a between ˜ N a ( T 6 -1) and ˜ N a ( T 6 ) , can take ˆ N b = ˜ N b ( T 6 ) for all b = a , and define the tuple ˆ N = ( ˆ N b ) b ∈ [ K ] , such that, (62) is bounded by,

Using (60) and the above upper bound, we have,

Induction: Note that at a given iteration N the algorithm can only see g ( ˜ µ ( N ) , ˜ N ( N )) . By (60), for N ≥ T 6 , g ( ˜ µ , ˜ N ) and g ( µ , ˜ N ) may have different signs only when ∣ ∣ ∣ g ( µ , ˜ N ( N )) ∣ ∣ ∣ ≤ C 1 N -3 α/ 8 . Based on this, we consider two cases.

̸

Case I: | g ( µ , ˜ N ( N )) | ≤ C 1 N -3 α/ 8 : We assume a ∈ [ K ] to be the arm pulled in iteration N +1 . Using the mean value theorem, we can find ˆ N a ∈ [ ˜ N a ( N ) , ˜ N a ( N +1) ] , can take ˆ N b = ˜ N b ( N ) for all b = a , and define the tuple ˆ N = ( ˆ N b ) b ∈ [ K ] , such that,

where (1) follows from (61).

Note that N ≥ T 6 ≥ M 4 ≥ M 42 . By the definition of M 42 , we have

̸

Case II: | g ( µ , ˜ N ( N )) | > C 1 N -3 α/ 8 : In this case g ( ˜ µ ( N ) , ˜ N ( N )) and g ( µ , ˜ N ( N )) have the same sign. Let arm a has been sampled from in iteration N +1 . Using the mean value theorem, we have ˆ N a ∈ [ ˜ N a ( N ) , ˜ N a ( N +1) ] , can take ˆ N b = ˜ N b ( N ) for all b = a , and define the tuple ˆ N = ( ˆ N b ) b ∈ [ K ] , such that,

We first consider the case when g ( µ , ˜ N ( N )) > 0 . After the algorithm sees g ( ˜ µ ( N ) , ˜ N ( N )) > 0 , it pulls the first arm. As a result, by (61) and (63), g ( µ , ˜ N ( · )) decreases in iteration N +1 atmost by C ′ 2 N -1 and atleast by C 2 N -1 . Now there can be two possibilities:

If g ( µ , ˜ N ( N + 1)) < 0 , we must have g ( µ , ˜ N ( N + 1)) ≥ -C ′ 2 N -1 . Since N ≥ T 6 ≥ M 4 ≥ M 43 , we have C ( N +1) -3 α/ 8 ≥ C ′ N -1 by the definition of M 43 . As a result,

If g ( µ , ˜ N ( N +1)) ≥ 0 , then g ( µ , ˜ N ( · )) has moved towards zero by atleast C 2 N -1 . Whereas, by iteration N +1 , the interval [ -CN -3 α/ 8 , CN 3 α/ 8 ] has reduced from both ends by

Since N ≥ T 6 ≥ M 4 ≥ M 44 , by the definition of M 44 , we have 3 Cα 8 ( N +1) -(1+ 3 α 8 ) < C 2 N -1 for every N ≥ T 6 . As a result, we can ensure g ( µ , ˜ N ( N +1)) ≤ C ( N +1) -3 α/ 8 at iteration N +1 .

In the other case, when g ( µ , ˜ N ( N )) < 0 , the algorithm sees g ( ˜ µ ( N ) , ˜ N ( N )) < 0 , and hence pulls some arm a ∈ [ K ] / { 1 } . As a result, by (61) and (63), g ( µ , ˜ N ( · )) increases in iteration N +1 atmost by C ′ 2 N -1 and atleast by C 2 N -1 . Then we apply the same argument as for the case g ( µ , ˜ N ( N )) > 0 , but by reversing the signs. Therefore, the inductive statement holds true for this case as well. Hence (59) stands proved.

T 6 has finite expectation: By Lemma G.9, we can have a constant C 3 > 0 , such that T 6 ≤ C 3 max { M 4 , T 5 } . As a result, since E µ [ T 5 ] < ∞ , we have E µ [ T 6 ] ≤ C 1 ( M 4 + E µ [ T 5 ]) < ∞ .

G.1.2 Closeness of the indexes

Lemma G.2 is a detailed version of Lemma 5.1 mentioned in Section 5, and is essential for proving closeness of the indexes under the allocations made by AT2 and IAT2 algorithms. Recall that T 6 is the random time defined in Proposition G.1 and satisfies E µ [ T 6 ] < ∞ .

Lemma G.2. For both AT2 and IAT2 algorithms, there exists constants M 5 ≥ 1 and C 1 > 0 independent of the sample paths, such that, for every N ≥ max { M 5 , T 6 } , if the algorithm picks an arm a ∈ [ K ] / { 1 } at iteration N , then it again picks arm a within the next ⌈ C 1 N 1 -3 α/ 8 ⌉ iterations.

Proof of Lemma G.2 is in Appendix G.6, and requires proving several technical lemmas. Some of those supporting lemmas involve arguments similar to the ones used for proving closeness of the indexes while the algorithm operates under an idealized fluid model (discussed in Section 4). In the rest of this section, we use Lemma G.2 to prove closeness of indexes for alternative arms in Proposition G.2.

Definition G.2. We define the random time T 7 = max { M 5 , T 6 } .

Note that E µ [ T 7 ] < ∞ , since E µ [ T 6 ] < ∞ .

Definition G.3. For every M ≥ 1 , define T 7 ,M = max { M,T 7 } , and T 8 ,M as the smallest iteration after T 7 ,M by which all the alternative arms in [ K ] / { 1 } have been picked atleast once by the algorithm.

Below we state a detailed version of Proposition G.2.

Proposition G.3. For every M ≥ 1 , we have E µ [ T 7 ,M ] < ∞ and E µ [ T 8 ,M ] < ∞ . Moreover, for every M ≥ 1 and N ≥ T 8 ,M , every pair of arms a, b ∈ [ K ] / { 1 } satisfy,

where the constant hidden in O ( · ) is independent of M and the sample path after T 7 .

Definition G.4. We define T 8 = T 8 , 1 , where T 8 , 1 is defined according to Proposition G.3.

By the defintion of T 8 above, Proposition G.2 follows trivially from Proposition G.3.

The following lemma helps us to bound the deviation of the empirical index I a ( N ) from the index I a ( N ) due to the noise in the estimates ˜ µ , for every alternative arm a ∈ [ K ] / { 1 } .

Lemma G.3. For a ∈ [ K ] / { 1 } and N ≥ T 0 , we have,

Proof. Proof of this lemma uses mean value theorem. For any arm a ∈ [ K ] / { 1 } , upon expanding the indexes,

where ˜ N 1 , ˜ N a , ˜ µ 1 , ˜ µ a , and ˜ x 1 ,a are evaluated at N . Since ˜ N 1 , ˜ N a ≤ N , the difference (64) is bounded above by,

Now considering the first term in the RHS, and applying mean value theorem, we get

where ˆ µ 1 , ˆ µ a , respectively, lie between ˜ µ 1 , µ 1 , and ˜ µ a , µ a , and ˆ x 1 ,a = ˜ N 1 ˆ µ 1 + ˜ N a ˆ µ a ˜ N 1 + ˜ N a . Using (7) and (8), all the partial derivatives in the above upper bound are O (1) for N ≥ T 0 . Therefore,

Following a similar procedure, we can argue using (7) and (8), that the partial derivatives of d ( ˜ µ j , ˜ x 1 ,j ) with respect to ˜ µ 1 and ˜ µ j are O (1) in magnitude. As a result, using the mean value theorem,

for N ≥ T 0 and completing the proof.

Proof of Proposition G.2: We have E µ [ T 7 ,M ] ≤ M + E µ [ T 7 ] < ∞ . By Lemma G.13, ˜ N a ( N ) = Θ ( N ) for N ≥ T 7 ,M . Hence, by the definition of T 8 ,M , there exists a constant C ′ > 0 independent of M , such that, for every M ≥ 1 , T 8 ,M ≤ C ′ T 7 ,M . As a result, E µ [ T 8 ,M ] ≤ C ′ E µ [ T 7 ,M ] < ∞ .

Note that T 7 , 1 = T 7 . Also, for every M ≥ 1 , T 8 ,M ≥ T 8 , 1 = T 8 ( T 8 is defined in Definition G.4). It is sufficient to prove the proposition for every N ≥ T 8 .

We now argue for the algorithms AT2 and IAT2 separately.

AT2: We consider any two alternative arms a, b ∈ [ K ] / { 1 } , and define the time τ a,b ( N ) as,

Note that N + τ a,b ( N ) must be before the iteration after N by which the algorithm has picked both a and b atleast once. By the definition of T 7 and T 8 , for every N ≥ T 8 , all alternative arms in [ K ] / { 1 } has been sampled from atleast once between iterations T 7 and N . Therefore, by Lemma G.2, we have τ a,b ( N ) = O ( N 1 -3 α/ 8 ) .

where the last step follows from Lemma G.3. Now,

Therefore, we have,

Therefore,

By Lemma G.3, we know | I a ( N ) -I b ( N ) | ≤ |I a ( N ) -I b ( N ) | + O ( N 1 -3 α/ 8 ) . Therefore, the above inequality implies,

Using mean value theorem, for j ∈ { a, b } , we have,

where ˆ N i ∈ [ ˜ N i ( N ) , ˜ N i ( N + τ a,b ( N )) ] for i = 1 , a, b .

We know,

where ˆ x 1 ,j = ˆ N 1 µ 1 + ˆ N j µ j ˆ N 1 + ˆ N j . Note that both the partial derivatives above are bounded from above by max { d ( µ 1 , µ a ) , d ( µ a , µ 1 ) } , and therefore O (1) . As a result, since τ a,b ( N ) = O ( N 1 -3 α/ 8 ) , we have,

Using (69) in (67), we get

IAT2: First we define the modified empirical index of every alternative arm a ∈ [ K ] / { 1 } using the notation I ( m ) a ( N ) as,

We define the time τ ( m ) a,b ( N ) as,

Note that, for every a ∈ [ K ] / { 1 } , I ( m ) a ( N ) differs from I a ( N ) by atmost log( N ) and I a ( N ) differs from I a ( N ) by atmost O ( N 1 -3 α/ 8 ) for N ≥ T 0 . Therefore,

Now N + τ ( m ) a,b ( N ) must be earlier than the iteration after N by which the algorithm has picked both a and b atleast once. Using the same argument as AT2, by Lemma G.2, we have τ ( m ) a,b ( N ) = O ( N 1 -3 α/ 8 ) . Also, following the same steps as AT2, by replacing the empirical index I with the modified empirical index I ( m ) for every alternative arm, we obtain,

Using the mean value theorem, since the parital derivatives of I a and I b with respect to ˜ N 1 , ˜ N a and ˜ N b are O (1) , we have

From the last two observations, we conclude

G.2 Convergence of algorithm to optimal proportions

In this appendix we prove a slightly detailed version of Proposition 3.1 from Section 3. In Proposition 3.1, we argue that the proportion of samples allocated by the algorithm converges to the optimal proportions for the instance, a.s. in P µ , as the no. of samples grows to ∞ .

For every M ≥ 1 , we use T 7 ,M and T 8 ,M as defined in Definition G.3 in Appendix G.1.2. Recall that T 7 = T 7 , 1 and T 8 = T 8 , 1 . By Proposition G.3, we have E µ [ T 8 ,M ] < ∞ , and T 8 ,M ≥ T 8 for every M ≥ 1 .

Recall that ω ⋆ is the unique optimal allocation according to Proposition 2.2, and ˜ ω ( N ) = ( ˜ ω a ( N ) : a ∈ [ K ]) with ˜ ω a ( N ) = ˜ N a ( N ) N is the algorithms allocation at iteration N . We now state a slightly detailed version of Proposition 3.1 from Section 3,

Proposition G.4. There exists constants C 1 > 0 and M 6 ≥ 1 depending on µ , α , and K such that, for every N ≥ T 8 ,M 6 and a ∈ [ K ] , we have

where ϵ ( µ ) is a constant depending only on µ and defined in Appendix B.

Detailed proof of Proposition G.4 is in Appendix G.2.2 and relies on using IFT.

Below we define the random times T good and T stable , which are mentioned in the statements of Proposition 3.1, 5.1, and Lemma 5.1 from the main body of the paper.

Definition G.5 ( T stable and T good ) . We define T good = T 7 ,M 6 and T stable = T 8 ,M 6 , where M 6 ≥ 1 is introduced in Proposition G.4.

Remark G.1. Note that, by definition, T good ≥ T 4 , T 6 . As a result, by Proposition G.1 and Lemma G.13, ∣ ∣ ∣ g ( µ , ˜ N ( N )) ∣ ∣ ∣ = O ( N -3 α/ 8 ) and ˜ N j ( N ) = Θ ( N ) for every j ∈ [ K ] and N ≥ T good .

Proof of Lemma 5.1: By the definition of T 7 ,M , T 8 ,M in Appendix G.1.2, and since T good = T 7 ,M 6 , T stable = T 8 ,M 6 , every alternative arm in [ K ] / { 1 } gets picked atleast once between the iterations T good and T stable . The other part of the statement of Lemma 5.1 follows from Lemma G.2 because T good ≥ T 7 .

Before proving Proposition G.4 in Appendix G.2.2, we find a tighter upper bound on the time to reach optimal proportion T stable in the following Appendix G.2.1. While doing this, we identify a similarity between the time to reach stabilty in fluid dynamics and that for the algorithm.

G.2.1 Bounding time to reach stability

Lemma G.4 gives an upper bound on the time to reach stability for the algorithmic allocations. We define ω ⋆ min = min a ∈ [ K ] / { 1 } ω ⋆ a .

According to the discussion in Section 4, if the fluid dynamics has state ˜ N ( T good ) at time T good , then it hits all the indexes and reaches stability by a time atmost T good ω ⋆ min . In Lemma G.4, we argue that, the algorithm also approximately reaches the optimal proportion ω ⋆ by atmost ≈ T good ω ⋆ min iterations.

Lemma G.4. For every M ≥ M 6 ( M 6 is a constant defined in the statement of Proposition G.4),

Proof. From Proposition G.4, it follows that, for every M ≥ M 6 and N ≥ T 8 ,M ≥ T 8 ,M 6 , we have,

Since T 8 ,M is the first iteration after T 7 ,M by which every alternative arm has been picked atleast once, we have some arm a ∈ [ K ] / { 1 } such that,

which implies

Moreover, we have

Now by (70), we have

Combining the last two observation, we have,

for every M ≥ M 6 .

Since T 8 ,M , T 7 ,M →∞ as M →∞ a.s. in P µ , we have,

G.2.2 Proving Proposition G.4

By Proposition G.1 and G.3, there exists a constant C > 0 independent of the sample paths, such that ˜ ω ( N ) = ( ˜ ω a ( N )) a ∈ [ K ] satisfies,

for all N ≥ T 8 a.s. in P µ .

Proof of Proposition G.4 relies on using the implicit function theorem. Before proving the proposition, we describe below the framework over which we apply the implicit function theorem. We define the following functions,

̸

where ω = ( ω a ) a ∈ [ K ] ∈ R K ≥ 0 , η = ( η a ) a ∈ [ K ] ∈ R K , I ∈ R , and for every a ∈ [ K ] / { 1 } , x 1 ,a ( ω 1 , ω a ) = ω 1 µ 1 + ω a µ a ω 1 + ω a and W a ( ω 1 , ω a ) = ω 1 d ( µ 1 , x 1 ,a ( ω 1 , ω a )) + ω a d ( µ a , x 1 ,a ( ω 1 , ω a )) .

Using the functions defined above, we define the vector valued function Ψ ( ω , I, η ) as follows,

Ψ maps tuples of the form ( ω , I, η ) ∈ R K ≥ 0 × R × R K to R K +1 .

Its easy to observe that for every ω = ( ω a : a ∈ [ K ]) ∈ R K ≥ 0 satisfying ∑ a ∈ [ K ] ω a = 1 , and I ∈ R , there is a unique η ∈ R K for which Ψ ( ω , I, η ) = 0 K +1 . We refer to the quantity max a ∈ [ K ] | η a | as the violation caused by the pair ( ω , I ) to the optimality conditions in (54).

By (54), all the alternative arms in [ K ] / { 1 } have equal normalized index under the optimal allocation ω ⋆ . Let I ⋆ = W a ( ω ⋆ 1 , ω ⋆ a ) for every a ∈ [ K ] / { 1 } . Then Proposition 2.2 implies ( ω ⋆ , I ⋆ ) is the unique tuple satisfying

To prove Proposition G.4 we need the two technical lemmas: Lemma G.5 and G.6. Let us define ∥ x ∥ ∞ = max a ∈ [ K ] | x a | for every x ∈ R K .

Lemma G.5 shows that, the set of allocations satisfying the optimality conditions in (54) upto a maximum violation of r > 0 shrinks to ω ⋆ as r decreases to zero. In Lemma G.6, we use Lemma G.5 and IFT to argue that if the perturbation vector η satisfies ∥ η ∥ ∞ ≤ η max , where η max > 0 is a constant depending only on µ , then there is a unique pair ( ω , I ) satisfying Ψ ( ω , I, η ) = 0 K +1 . Moreover, the function mapping a perturbation vector η ∈ [ -η max , η max ] K to the unique pair ( ω , I ) solving Ψ ( ω , I, η ) = 0 K +1 is Lipschitz continuous.

It is now easy to see Proposition G.4 follows from Lemma G.5 and G.6. By (71), the violation caused by the algorithmic allocation ˜ ω ( N ) to the optimality conditions in (54) converges to zero uniformly at a rate O ( N -3 α/ 8 ) . We wait for sufficiently many iterations such that, the violation becomes smaller than η max . Then using Lipschitzness of the allocation as a function of perturbation (proven in Lemma G.6), we have ∥ ˜ ω ( N ) -ω ⋆ ∥ ∞ = O ( N -3 α/ 8 ) .

Lemma G.5. For every r ≥ 0 , we define the quantity,

The following statements are true about the mapping r ↦→ dist ( ω ⋆ , r ) ,

dist ( ω ⋆ , 0) = 0 ,

dist ( ω ⋆ , r ) is non-decreasing in r , and

lim r → 0 dist ( ω ⋆ , r ) = 0 .

Proof. Statement 1: Statement 1 follows directly from the fact that ( ω ⋆ , I ⋆ ) is the unique tuple satisfying Ψ ( ω ⋆ , I ⋆ , 0 K ) = 0 K +1 , as proven in Proposition 2.2.

Statement 2: Follows directly from the definition of dist ( ω ⋆ , r ) .

Statement 3: By statement 2, lim r → 0 dist ( ω ⋆ , r ) exists and is non-negative. We consider a contradiction to statement 3 and assume that lim r → 0 dist ( ω ⋆ , r ) = d > 0 .

Since r → dist ( ω ⋆ , r ) is non-decreasing, we can construct a decreasing sequence { r n } n ≥ 1 such that, for every n ≥ 1 , r n > 0 , dist ( ω ⋆ , r n ) ≥ d , and lim n →∞ r n = 0 . As a result, using the definition of dist ( ω ⋆ , r ) , we have a sequence of tuples { ( ω n , I n , η n ) } n ≥ 1 , such that,

such that,

ω ( 0 K ) = ω ⋆ , I ( 0 K ) = I ⋆ , and

for every η ∈ ( -δ 0 , δ 0 ) K , ( ω ( η ) , I ( η )) is the unique tuple in R K ≥ 0 × R to satisfy,

Since Ψ K +1 ( ω n ) = 0 for every n ≥ 1 , the whole sequence ( ω n ) n ≥ 1 lies in the set

which is compact with respect to the norm ∥·∥ ∞ .

Let for every n ≥ 1 and a ∈ [ K ] , ω a,n and η a,n be, respectively, the a -th component of the vectors ω n and η n . For every n ≥ 1 and a ∈ [ K ] / { 1 } , we have I n = W a ( ω 1 ,n , ω a,n ) -η a,n . Since W a ( · , · ) always lies in the interval [0 , d ( µ 1 , µ a ) + d ( µ a , µ 1 )] and | η a,n | ≤ r n , we have

By our assumption, we already have r n → 0 , which also implies, the sequence r n is bounded from above. As a result, I n is also bounded. Therefore, we can have a convergent subsequence { ( ω n k , I n k ) } k ≥ 1 , with limits ω n k → ω ′ and I n k → I ′ in the ∥·∥ ∞ -norm.

For every k ≥ 1 , we have Ψ ( ω n k , I n k , η n k ) = 0 K +1 . As a result, using the continuity of Ψ ( · ) with respect to its arguments, we have Ψ ( ω ′ , I ′ , 0 K ) = 0 K +1 , implying ω ′ is an optimal allocation for the instance µ .

̸

Hence, our assumption lim inf n →∞ max {∥ ω n -ω ⋆ ∥ ∞ , | I n -I ⋆ |} ≥ d implies max {∥ ω ′ -ω ⋆ ∥ ∞ , | I ′ -I ⋆ |} ≥ d > 0 , which further implies ω ′ = ω ⋆ . As a result, the instance µ has two distinct optimal allocations ω ′ and ω ⋆ , which contradicts Proposition 2.2.

Lemma G.6. There exists η max > 0 depending only on the instance µ , such that the following statements are true,

For every η ∈ [ -η max , η max ] K , there exists a unique tuple ( ω , I ) ∈ R K ≥ 0 × R which satisfies, Ψ ( ω , I, η ) = 0 K +1 .

For every η ∈ [ -η max , η max ] K , we call the unique tuple mentioned in statement 1 as ( ω ( η ) , I ( η )) . Then the function

is L -Lipschitz, for some L > 0 depending on the instance µ .

Proof. By Proposition 2.2, we know that the optimal allocation ω ⋆ is the unique allocation satisfying,

for some I ⋆ > 0 . Note that Ψ ( ω , I, η ) = Φ ( ω , I, η , 1) for every tuple ( ω , I, η ) , where Φ is the function defined in Appendix C. By statement 3 of Lemma C.1, the Jacobian ∂ Ψ ∂ ( ω ,I ) of the function Ψ ( ω , I, η ) is invertible at the tuple ( ω ⋆ , I ⋆ , 0 K ) .

Therefore, applying the Implicit function theorem, we can find δ 0 , δ 1 > 0 , and continuously differentiable functions

.

By statement 3 of Lemma G.5, we can find a δ 2 > 0 such that, dist ( ω ⋆ , r ) < δ 1 for r ∈ [0 , δ 2 ] . We define η max = min { δ 0 2 , δ 2 } .

By the definition of dist ( ω ⋆ , · ) , for every η ∈ [ -η max , η max ] K , if a tuple ( ω , I ) satisfies Ψ ( ω , I, η ) = 0 K +1 , then it also satisfies max {∥ ω -ω ⋆ ∥ ∞ , | I -I ⋆ |} < δ 1 .

On the other hand, by IFT, since η max < δ 0 , ( ω ( η ) , I ( η )) is the only such tuple possible. Therefore, for every η ∈ [ -η max , η max ] K , ( ω ( η ) , I ( η )) is the unique element in R K ≥ 0 × R such that Ψ ( ω ( η ) , I ( η ) , η ) = 0 K +1 . This proves the first statement of Lemma G.6.

Since ω ( · ) , I ( · ) is continuously differentiable in ( -δ 0 , δ 0 ) K , every component of this mapping must be L -Lipschitz for some L > 0 in [ -δ 0 2 , δ 0 2 ] K equipped with ∥·∥ ∞ -norm. We can take L to be the maximum of the ∥·∥ 1 -norm of the gradients of different components of ( ω ( · ) , I ( · )) over the set [ -δ 0 2 , δ 0 2 ] K . Since the gradients are all continuous, their ∥·∥ 1 -norm must be bounded in a compact set like [ -δ 0 2 , δ 0 2 ] K , and hence L < ∞ . Therefore, the second part of Lemma G.6 follows from our assumption η max ≤ δ 0 2 .

We now proceed on proving Proposition G.4.

Proof of Proposition G.4: Recall that in Section 5, for every a ∈ [ K ] / { 1 } , we defined the normalized index as H a ( N ) = I a ( N ) N .

Taking H ( N ) = H 2 ( N ) = I 2 ( N ) N , let ˜ η ( N ) be the unique η ∈ R K to satisfy, Ψ ( ˜ ω ( N ) , H ( N ) , η ) = 0 K +1 .

Note that for every a ∈ [ K ] / { 1 } , we have W a ( ˜ ω 1 ( N ) , ˜ ω a ( N )) = H a ( N ) . As a result, by (71), we have ∥ ˜ η ( N ) ∥ ∞ ≤ CN -3 α/ 8 for all N ≥ T 8 .

Now we pick M 6 ≥ 1 large enough, such that,

where η max is introduced in Lemma G.6. We define T stable = T 8 ,M 6 . Note that T stable ≥ T 6 ≥ T 0 . As a result, by the definition of T 0 in Appendix G.3, we have max a ∈ [ K ] | ˜ µ a ( N ) -µ a | ≤ ϵ ( µ ) N -3 α/ 8 for every N ≥ T stable .

Now, by (71), for N ≥ T stable , the allocations ˜ ω ( N ) satisfies, Ψ ( ˜ ω ( N ) , H ( N ) , ˜ η ( N )) = 0 K +1 with

As a result, by Lemma G.6, we have

where ω a ( · ) is the a -th component of the vector valued function ω ( · ) introduced in Lemma G.6.

By Lemma G.6, for every a ∈ [ K ] , ω a ( · ) is L -Lipschitz in [ -η max , η max ] K equipped with ∥·∥ ∞ -norm. As a result, for N ≥ T stable , we have,

Taking C 1 = LC , we have the desired result.

G.3 T 0 has finite expectation

In Section 5, we introduced the random time T 0 as,

where ϵ ( µ ) is a positive constant defined in Appendix B and depends only on the instance µ .

Lemma G.7. The random time T 0 satisfies E µ [ T 0 ] < ∞ and hence T 0 < ∞ a.s. in P µ .

Proof. To avoid notational clutter, let P = P µ and ϵ = ϵ ( µ ) . Then for any N ,

where ˆ µ a,t denotes the empirical mean of t i.i.d. samples drawn from the a -th arm, and the last step follows from statement 2 of Proposition G.5, which says ˜ N a ( N ) ≥ N α -C 1 for some constant C 1 > 0 .

Using Chernoff's bound (like in the proof of Lemma 19 of [11]), we have,

Using (6), we have a constant C 2 > 0 depending on the instance µ and such that,

Therefore, we have,

Therefore,

where the constant hidden in Ω ( · ) depends only on µ . Using the obtained upper bound,

Note that the series on the RHS is convergent for any α ∈ (0 , 1) . Therefore E µ [ T 0 ] < ∞ .

G.4 Properties of exploration

In the following discussion, the set of iterations in which the algorithm does exploration is defined as all iterations N where min a ∈ [ K ] ˜ N a ( N -1) < N α (which is equivalent to having V N = ∅ , where V N denotes the set of starved arms at iteration N , and is defined in Section 3). We define the epoch of exploration at some arbitrary iteration as follows,

̸

Definition G.6. If the algorithm does exploration at iteration N , the epoch of exploration at N is the maximum no. of consecutive iterations including N in which the algorithm has done exploration. More precisely, if N 1 and N 2 are, respectively, defined as,

then the epoch of exploration at iteration N is N 2 -N 1 +1 .

Proposition G.5. The following statements are true:

For every iteration which is an exploration, the epoch of exploration at that iteration is upper bounded by a constant depending on K and α . We denote this constant using T epoch .

There exists a constant C depending on K and α such that , over every sample path, we have

As a result, ˜ N a ( N ) = Ω ( N α ) for every arm a ∈ [ K ] .

There exists a M depending on K and α such that, every epoch of exploration starting after iteration M has length atmost K , and every arm can get pulled atmost once in that epoch. We call the constant M as M explo .

If an epoch of exploration starts from some N ≥ M explo, then the next epoch of exploration doesn't start before another Θ ( N 1 -α ) iterations.

Let ˆ N ≥ M explo be such that, ˆ N is an exploration. Define the following sequence, N 0 = ˆ N , and for k ≥ 1 ,

Then N k = ˆ N + Ω ( k 1 /α ) . In other words, for any N ≥ M explo , the k -th epoch of exploration after iteration N starts after N + Ω ( k 1 /α ) iterations.

For any N ≥ M explo + T epoch +1 and T ≥ 1 , the no. of epochs of exploration intersecting with the set of iterations { N,N +1 , N +2 , . . . , N + T } is O ( T α ) .

Proof. Statement 1: Let the algorithm does exploration at iteration N . We can always choose N in such a way that, iteration N -1 was not an exploration, by choosing N to be the iteration at which an epoch begins. If the epoch of exploration starting at iteration N continues till iteration N + t , i.e. , the iterations N,N +1 , . . . , N + t are exploration, then,

where (1) follows from the fact that, min a ∈ [ N ] ˜ N a ( · ) increments by atleast 1 over every K consecutive iterations in an epoch of exploration, and (2) from the fact that iteration N -1 is not an exploration. From the above inequality, we have,

Note that, N → ( N + t ) α -( N -1) α is decreasing in N . Hence RHS ≤ (1 + t ) α ≤ 1 + t α ≤ 2 t α (since t ≥ 1 ). Therefore, we have,

Therefore every epoch of exploration is atmost (2 K ) 1 / (1 -α ) iterations long.

Statement 2: In the following discussion we use [ i : j ] for a pair of integers i < j to denote the set { i, i +1 , i +2 , . . . , j } . We consider only those iterations where min a ∈ [ K ] ˜ N a ( N -1) < N α . By statement 1, if we consider N 1 ≤ N ≤ N 2 such that, N 1 , N 1 +1 , . . . , N 2 -1 , N 2 are all explorations and N 1 -1 , N 2 +1 are not explorations, then we must have, N 2 -N 1 ≤ T epoch. As a result, we have,

Now, the slowest rate at which min a ∈ [ K ] ˜ N a ( N ) can grow while iterations N ∈ { N 1 , N 1 + 1 , . . . , N 2 -1 , N 2 } is if the algorithm pulls the K arms consecutively in those iterations. Therefore, we have,

Using this, we have,

Since iteration N 1 -1 is not an exploration, we have

Using this, the upper bound becomes,

where C depends only on K and α .

Hence we get,

Statement 3: If the epoch of exploration starts from T and continues for more than K iterations, note that min a ∈ [ K ] ˜ N a ( · ) gets incremented by atleast 1 during the iterations T, T +1 , . . . , T + K . As a result, we have,

Since iteration T -1 is not an exploration, we have ˜ N a ( T -1) ≥ ˜ N a ( T -2) ≥ ( T -1) α . Similarly, since iteration T + K is an exploration, min a ∈ [ K ] ˜ N a ( T + K -1) < ( T + K ) α . Using these two observations, we have,

Therefore,

Let M explo = ( α ( K +1)) 1 / (1 -α ) +2 . Then from the above argument, if T ≥ M explo, the epoch of exploration starting at T will last for at most K iterations.

Let us assume that, the epoch begining from some T i ≥ M explo lasts till iteration T f . Therefore, we have,

Using this,

We argued earlier that T f -T i ≤ K . We also have, T i -1 ≥ M explo -1 ≥ ( α ( K +1)) 1 / (1 -α ) +1 . Using this observations, we get

Since min a ∈ [ K ] ˜ N a ( T f ) ≤ min a ∈ [ K ] ˜ N a ( T f -1)+1 and min a ∈ [ K ] ˜ N a ( T i -2) ≤ min a ∈ [ K ] ˜ N a ( T i -1) , we obtain,

implying min a ∈ [ K ] ˜ N a ( T f ) -min a ∈ [ K ] ˜ N a ( T i -1) ≤ 1 , which is possible only if every arm is pulled at most once in iterations T i , T i +1 , . . . , T f -1 , T f .

Statement 4: Let an epoch of exploration starts from N ≥ M explo and the next epoch starts from N + T for some T ≥ 1 . The epoch starting from N continues till atmost min { N + K, N + T -2 } by statement 3. Moreover in that epoch, every arm gets pulled atmost once and min a ∈ [ K ] ˜ N a ( · ) increments by 1 . Therefore,

Since iteration N -1 is not an exploration, we have, min a ∈ [ K ] ˜ N a ( N -1) ≥ min a ∈ [ K ] ˜ N a ( N -2) ≥ ( N -1) α . Since iteration N + T is an exploration, we have min a ∈ [ K ] ˜ N a ( N + T -1) < ( N + T ) α . Using these in the above inequality, we obtain,

which implies T ≥ 1 α ( N -1) 1 -α -1 = Θ ( N 1 -α ) .

Statement 5: Using statement 4, we can have a constant C 1 such that,

for k ≥ 1 , where N 0 = 1 . We now inductively argue that, there exists some constant C 2 independent of k and N such that, N k ≥ N + C 2 k 1 /α for k ≥ 1 .

For k = 1 , we choose C 2 ≤ 1 .

Now for some k ≥ 2 , if N ≥ N + C ( k -1) 1 /α , we have,

Upon choosing C 2 ≤ ( αC 1 2 1 α -1 ) 1 /α , since we already have k ≥ 2 , we obtain

Therefore, N k ≥ N + C 2 k 1 α .

Therefore, choosing C 2 = min { 1 , ( αC 1 2 1 α -1 ) 1 α } , we have N k ≥ N + C 2 k 1 α for all k ≥ 1 .

Statement 6: If N ≥ M explo + T epoch + 1 and T ≥ 1 , every epoch of explorations intersecting with the iterations { N, N +1 , . . . , N + T -1 , N + T } has length atmost K (by statement 3). Hence, every such epoch must have started on or after iteration N -K . Let N 0 be the time when the first such epoch has started and the sequence ( N k ) k ≥ 1 be defined similar to statement 5. Then N 0 ≥ N -K and N k ≥ N 0 + C 2 k 1 α ≥ N -K + C 2 k 1 α . Now, if the k -th epoch starting after N 0 intersects with the iterations { N, N +1 , . . . , N + T -1 , N + T } , then,

Definition G.7. We define the constant T explo = M explo + T epoch +1 , where the constants M explo, and T epoch are defined in Proposition G.5.

Lemma G.8. For N ≥ T explo and T ≥ 1 , the no of times an arm a ∈ [ K ] is pulled for exploration by the algorithm during iterations N, N +1 , . . . , N + T -1 , N + T is O ( T α ) .

Proof. By statement 3 of Proposition G.5, for N ≥ T explo and T ≥ 1 , every epoch of exploration intersecting with the set of iterations N, N +1 , N +2 , . . . , N + T is of length atmost K . In every such epoch, every arm is pulled atmost once. As a result, no. of times an arm is pulled during the iterations N, N +1 , . . . , N + T is upper bounded by the no. of epoch of iterations intersecting with the set { N, N +1 , . . . , N + T -1 , N + T } . The later quantity is O ( T α ) by statement 6 of Proposition G.5. Hence the lemma stands proved.

G.5 Technical lemmas related to algorithmic allocations

In this appendix we prove several properties about the anchor function ( g ) and the algorithmic allocations ˜ N ( N ) = ( ˜ N a ( N ) : a ∈ [ K ]) . We exploit the results proven in Appendix B and G.4 to prove that the following properties hold after a random time of finite expectation,

̸

if the algorithm has g = 0 at some iteration N , then g crosses the value zero withing an O ( N ) iterations, where the constant hidden in O ( · ) is independent of the sample paths (in Lemma G.9), and

every arm a ∈ [ K ] has ˜ N a ( N ) = Θ ( N ) samples (in Lemma G.13).

Each of the properties stated above are used extensively in the proofs of Proposition G.1 and G.2.

Definition G.8. We define the random time T 1 = max { T 0 , T explo } (where T explo and T 0 are defined, respectively, in Appendix G.4 and G.3).

We can have g far from the value zero at iteration T 1 . g will still be finite at T 1 because of exploration. Lemma G.9 bounds the no. of iterations the algorithm takes to reach the value zero.

̸

̸

Lemma G.9 ( Upper bound to the time to reach g = 0 ) . If g ( ˜ µ ( N ) , ˜ N ( N )) = 0 at iteration N ≥ T 1 , and

U = min { t > 0 ∣ ∣ ∣ g ( ˜ µ ( N + t ) , ˜ N ( N + t ) ) and g ( ˜ µ ( N ) , ˜ N ( N ) ) have opposite signs } , then there exists a constant C 1 > 0 independent of the sample paths such that U ≤ C 1 N .

Proof. Using (10), for N ≥ T 1 , we have,

̸

We now consider two situations separately,

Case I: g ( ˜ µ ( N ) , ˜ N ( N ) ) > 0 : We know that, there is a constant D 1 > 0 , such that,

for every t ≥ 1 . Now for t < U , the algorithm selects an alternative arm from [ K ] / { 1 } only while exploring. Therefore, using statement 6 of Proposition G.5, we have a constant c 1 > 0 such that, ˜ N 1 ( N + t ) ≥ t -c 1 t α . Hence,

Since g ( ˜ µ ( N + U -1) , ˜ N ( N + U -1) ) ≥ 0 , RHS of the above inequality is non-negative at t = U -1 . After some algebraic manipulation, this implies,

Since LHS of the above inequality is linear in U , we can find a constant C 11 such that U ≤ C 11 N .

Case II: g ( ˜ µ ( N ) , ˜ N ( N ) ) < 0 : Using (72), we have a constant D 2 such that,

for all t ≥ 1 . Now for t < U , the algorithm pulls arm 1 only for exploration. As a result, using statement 6 of Proposition G.5, we have, a constant c 1 > 0 such that ˜ N 1 ( N + t ) ≤ N + c 1 t α . Using this,

for t < U . Since g ( ˜ µ ( N + U -1) , ˜ N ( N + U -1)) ≤ 0 , the RHS of the above inequality is not positive at t = U -1 . After some algebraic manipulation, this implies,

Again the LHS of the above inequality is linear in U . As a result, we can find a constant C 12 such that, U ≤ C 12 N .

We take C 1 = max { C 11 , C 12 } and have U ≤ C 1 N .

Lemma G.10, G.11, and G.12 are necessary for proving Lemma G.13, which says ˜ N a ( N ) = Θ ( N ) for every arm a ∈ [ K ] , after a random time of finite expectation. This property is essential for proving Proposition G.1 and G.2 stated earlier.

Lemma G.10. There exists constants γ 1 , γ 2 ∈ (0 , 1) independent of the sample path, such that, for N ≥ T 1 , whenever g ( ˜ µ ( N ) , ˜ N ( N ) ) and g ( ˜ µ ( N +1) , ˜ N ( N +1) ) have opposite signs, we have,

˜ N 1 ( N ) ≥ γ 1 N , and

Proof. For N ≥ T 1 , we have constants D 1 , D 2 > 0 such that,

We consider only the situation where g ( ˜ µ ( N ) , ˜ N ( N ) ) ≥ 0 and g ( ˜ µ ( N +1) , ˜ N ( N +1) ) ≤ 0 . Extending this to the other case follows similar argument.

From g ( ˜ µ ( N +1) , ˜ N ( N +1)) ≤ 0 , we have,

Since ˜ N 1 ( N ) ≥ 1 (for exploration), we have ˜ N 1 ( N + 1) ≤ ˜ N 1 ( N ) + 1 ≤ 2 ˜ N 1 ( N ) . Therefore, using (73) we get ˜ N 1 ( N ) ≥ 1 2 (1 + D -1 / 2 1 ) -1 N and we can take γ 1 = 1 2 (1 + D -1 / 2 1 ) -1 .

Similarly from, g ( ˜ µ ( N ) , ˜ N ( N )) ≥ 0 , we have,

Using (74), we have, ∑ a ∈ [ K ] / { 1 } ˜ N a ( N ) ≥ (1 + D 1 / 2 2 ) -1 N . This further implies

Hence we can take γ 2 = 1 K -1 (1 + D 1 / 2 2 ) -1 .

Lemma G.11. There exists constants M 1 ≥ 1 , d max ∈ [1 , ∞ ) and d min ∈ (0 , 1] independent of the sample path, such that, if T 2 is the time at which g crosses zero after the iteration max { M 1 , T 1 } , then, for N > T 2 , we have:

̸

Moreover, we have E µ [ T 2 ] < ∞ .

Proof. Let,

̸

Also for every M ≥ 1 , we define T 1 ,M = max { M,T 1 } and T 2 ,M as the iteration at which g crosses zero for the first time after the iteration T 1 ,M , i.e. ,

Existence of d max : If g ( ˜ µ ( N ) , ˜ N ( N )) ≤ 0 , then D ( ˜ µ ( N ) , ˜ N ( N )) is bounded above trivially by 1 .

Otherwise if g ( ˜ µ ( N ) , ˜ N ( N )) > 0 for some N > T 2 ,M , let T = max { t ≤ N | g ( ˜ µ ( t ) , ˜ N ( t )) ≤ 0 } i.e. the time before iteration N when g has crossed the level zero. As a result, T ≥ T 2 ,M (by definition of T 2 ,M ), and by Lemma G.10, ˜ N 1 ( T ) ≥ γ 1 T for some γ 1 ∈ (0 , 1) . Let S = N -T . During iterations T +1 , . . . , T + S , since g is positive, the algorithm pulls alternative arms in [ K ] / { 1 } only for exploration. As a result, using statement 6 of Proposition G.5, we have

for some c 1 > 0 . Using (72), we have a constant D 2 > 0 such that,

We take M 11 = -2 γ 1 min S ≥ 0 ( S -c 1 S α ) . It is easy to observe that for T ≥ M 11 and S ≥ 0 , the function of T, S in the RHS is bounded above. Therefore, we take,

Hence, if M ≥ M 11 , we have g ( ˜ µ ( N ) , ˜ N ( N )) ≤ d max for every N > T 2 ,M .

Existence of d min : If g ( ˜ µ ( N ) , ˜ N ( N )) ≥ 0 , we have 1 has the trivial lower bound. Otherwise, if g ( ˜ µ ( N ) , ˜ N ( N )) < 0 , we define T = max { t ≤ N | g ( ˜ µ ( t ) , ˜ N ( t )) ≥ 0 } and S = N -T . As a result, T ≥ T 2 ,M (by definition of T 2 ,M ). By Lemma G.10, we have ˜ N 1 ( T ) ≤ (1 -γ 2 ) N for some γ 2 ∈ (0 , 1) . Also, the algorithm pulls the first arm only for exploration during the iterations T +1 , T +2 , . . . , T + S , since g is negative. Therefore, using statement 6 of Proposition G.5, we have, ˜ N 1 ( T + S ) ≤ ˜ N 1 ( T ) + c 1 S α ≤ (1 -γ 2 ) T + c 1 S α . By (72), we have D 1 > 0 such that,

Let M 12 = -2 γ 2 min S ≥ 0 ( S -c 1 S α ) . The function of T, S in the RHS is bounded below by a positive constant for T ≥ M 12 and S ≥ 0 . Let,

Then for M ≥ M 12 , we have g ( ˜ µ ( N ) , ˜ N ( N )) ≥ d min > 0 for every N > T 2 ,M .

Now, upon taking M 1 = max { M 11 , M 12 } , and T 2 = T 2 ,M 1 , the lemma follows. By Lemma G.9, we have a constant C 1 > 0 such that, T 2 ≤ C 1 max { M 1 , T 1 } . As a result, E µ [ T 2 ] ≤ C 1 ( M 1 + E µ [ T 1 ]) < ∞ .

Using a technique similar to the one used in the proof of Lemma G.10 using (72), Lemma G.11 implies the following corollary. We define the random time T 2 as the one introduced in Lemma G.11.

Corollary G.1. There exists constants β 1 , β 2 ∈ (0 , 1) independent of the sample paths, such that, for every N > T 2 ,

˜ N 1 ( N ) ≥ β 1 N , and

Lemma G.12. There exists constants γ > 0 and M 2 ≥ 1 independent of the sample path, such that, for N > max { M 2 , T 2 } +1 , if the algorithm pulls some arm a ∈ [ K ] / { 1 } , then for every alternate arm b ∈ [ K ] / { 1 , a } , ˜ N b ( N ) ≥ γ ˜ N a ( N ) .

Proof. We consider two separate cases.

Case I: Iteration N is an exploration: Then we have a constant C 1 (introduced in statement 2 of Proposition G.5) such that,

We also have ˜ N a ( N ) ≥ N α -C 1 . Now let M 2 ≥ 2 be chosen such that, M α 2 > 3 C 1 +2 . As a result, for N ≥ max { M 2 , T 2 } we have, ˜ N a ( N ) ≥ N α -C 1 ≥ 2(1 + C 1 ) . This together with (75) implies, ˜ N b ( N ) ≥ 1 2 ˜ N a ( N ) .

Case II: Iteration N is not an exploration: We consider the AT2 (1) and IAT2 (2) algorithms separately.

Case II.I AT2 Algorithm: If iteration N is not an exploration, we have, I a ( N -1) ≤ I b ( N -1) . Using (11), for N ≥ max { M 2 , T 2 } , we have constants C 2 and C 3 such that,

As a result, we have,

where, for (1), we first note that N > T 2 +1 . As a result, using Corollary (G.1), ˜ N 1 ( N -1) ≥ β 1 ( N -1) ( β 1 ∈ (0 , 1) is the constant introduced in Corollary G.1), and on the other hand ˜ N 1 ( N -1) + ˜ N b ( N -1) ≤ 2( N -1) . Hence, we get

Again using statement 2 of Proposition G.5, ˜ N a ( N ) ≥ N α -C 1 ≥ M α 2 -C 1 ≥ 2 (since M α 2 ≥ 3 C 1 +2 ). As a result,

Taking γ = min { 1 2 , β 1 C 2 4 C 3 } , we have the desired result for AT2 algorithm.

Case II.II IAT2 Algorithm: In this case, we have,

Using (11), we have constants C 2 and C 3 such that,

As a result, using the same arguments used for the AT2 algorithm in Case II.I, we can conclude,

for N > max { M 2 , T 2 } +1 .

Now taking γ = min { 1 2 , β 1 4 ( C 2 C 3 + 1 e ) -1 } , we have the desired result for IAT2 algorithm.

Definition G.9. We define the random time T 3 = max { M 2 , T 2 } + 2 , where M 2 is the constant introduced in Lemma G.12, and T 2 is the random time defined in Lemma G.11.

Definition G.10. We define the random time T 4 as T 4 = 1 β 2 ( T 3 + 1) , where β 2 ∈ (0 , 1) is the constant introduced in Corollary G.1.

Note that E µ [ T 3 ] < ∞ , since E µ [ T 2 ] < ∞ . For the same reason, by Definition G.10, T 4 satisfies E µ [ T 4 ] < ∞ since E µ [ T 3 ] < ∞ .

Lemma G.13 ( Sufficient sampling ) . For every N ≥ T 4 , we have ˜ N a ( N ) = Θ ( N ) for every a ∈ [ K ] .

Proof. For a = 1 and every N ≥ T 4 , we have ˜ N 1 ( N ) = Θ ( N ) by Corollary G.1.

̸

Otherwise for a = 1 and every N ≥ T 4 , we define A ′ N = arg max b ∈ [ K ] / { 1 } ˜ N b ( N ) . By Corollary G.1,

Therefore, arm A ′ N must have been pulled by the algorithm somewhere between iterations T 3 and N . Let us define the time N ′ to be the last time before N when A ′ N was pulled. Then N ′ ≥ T 3 > max { M 2 , T 2 } +1 . As a result, using Lemma G.12, we have,

where (i) follows by definition of N ′ . Since N ≥ N ′ , we have ˜ N a ( N ) ≥ ˜ N a ( N ′ ) . As a result, we obtain, for every a ∈ [ K ] / { 1 } and N ≥ T 4 ,

where β 2 ∈ (0 , 1) is the constant mentioned in Corollary G.1. Hence we have ˜ N a ( N ) = Ω ( N ) for N ≥ T 4 and the lemma follows.

G.6 Proving Lemma G.2

In this appendix, we prove Lemma G.2 from Appendix G.1.2. For improved clarity, we reiterate Lemma G.2 from Appendix G.1.2.

Statement of Lemma G.2. For both AT2 and IAT2 algorithms, there exists constants M 5 ≥ 1 and C 1 > 0 independent of the sample paths, such that for every N ≥ max { M 5 , T 6 } , if the algorithm picks an arm a ∈ [ K ] / { 1 } at iteration N , then it again picks arm a within the next ⌈ C 1 N 1 -3 α/ 8 ⌉ iterations.

For every arm a ∈ [ K ] , iteration N ≥ 1 and R ≥ 1 , we define the following quantities,

For proving Lemma G.2, we need the three technical lemmas: Lemma G.14, G.15 and G.16.

Lemma G.14. For N ≥ T 6 ( T 6 is the random time defined in the statement of Proposition G.1 and it satisfies E µ [ T 6 ] < ∞ ), we have,

Proof. In the proof, for readability, we use b to denote b ( N,R ) . Also, for every arm a ∈ [ K ] , we use ∆ ˜ N a to denote ∆ ˜ N a ( N,R ) .

By Proposition G.1, we have a constant C > 0 independent of the sample paths, such that for N ≥ T 6 ,

Therefore, applying the mean value theorem, we have,

where ˆ N = ( ˆ N a ) a ∈ [ K ] , with ˆ N a ∈ [ ˜ N a ( N ) , ˜ N a ( N + R ) ] for every a ∈ [ K ] .

Now expanding the partial derivatives of g with respect to N a 's for a ∈ [ K ] , we get

where, f ( µ , a, ˆ N ) were defined in (13) of Appendix B. Letting

̸

for every arm j ∈ [ K ] / { 1 } , we have,

̸

We know from (14) of Appendix B that f ( µ , a, N ) = Θ ( N a N 1 ( 1 + N a N 1 ) 2 ) . As a result,

̸

By Lemma G.13, we have ˆ N a ≥ ˜ N a = Θ ( N ) and ˆ N 1 ≤ N + R for N ≥ T 6 . As a result, we get,

̸

Putting this in (78), we obtain

̸

which further implies,

̸

̸

Let b = arg max a =1 ∆ ˜ N a ˜ N a . The above inequality implies,

Now multiplying both sides by ˆ N 1 /∆ ˜ N b , we get,

(76) follows from the above inequality upon observing that, ˆ N 1 ≤ ˜ N 1 + R and ˜ N b = Ω ( N ) (by Lemma G.13).

Lemma G.15. There exists constants M 51 ≥ 1 , D 1 , D 2 > 0 independent of the sample paths such that, for N ≥ max { M 51 , T 6 } , we have ⌈ D 1 N 1 -3 α/ 8 ⌉ < N , and for all R ∈ {⌈ D 1 N 1 -3 α/ 8 ⌉ , ⌈ D 1 N 1 -3 α/ 8 ⌉ +1 , . . . , N } , we have ∆ ˜ N b ( N,R ) ( N,R ) ≥ D 2 R .

Proof. To simplify notation, we adopt b to denote b ( N,R ) and, for every arm j ∈ [ K ] , we use ∆ ˜ N j to denote ∆ ˜ N j ( N,R ) .

By (76) of Lemma G.14, there exists a constant D 3 > 0 independent of the sample paths, such that, for all N ≥ T 6 and R ∈ { 1 , 2 , . . . , N } ,

Since ˜ N 1 ( N ) and ˜ N b ( N ) are both Θ ( N ) (by Lemma G.13), we have a constant D 4 > 0 such that, ˜ N 1 ( N ) ˜ N b ( N ) + D 3 ≤ D 4 for every N ≥ T 6 . Using this in the above inequality, we get,

We take D 1 = 2 D 3 and M 51 ≥ 1 to be the smallest number such that, every N ≥ M 51 satisfies ⌈ D 1 N 1 -3 α/ 8 ⌉ < N .

Now if N ≥ max { M 51 , T 6 } and R ∈ {⌈ D 1 N 1 -3 α/ 8 ⌉ , ⌈ D 1 N 1 -3 α/ 8 ⌉ +1 , . . . , N } , from (79) we get,

̸

By definition of b , we have,

Again, since ˜ N a ( N ) = Θ ( N ) for every a ∈ [ K ] (by Lemma G.13), there exists a constant D 5 > 0 independent of the sample paths such that,

As a result,

Now taking D 2 = 1 2(1+( K -2) D + D ) > 0

5 4 , we have the desired conclusion.

Lemma G.16. For AT2 (1) and IAT2 (2) algorithms, there exists constants C 3 , C 4 > 0 independent of the sample paths, such that, for N ≥ T 6 and R ∈ { 1 , 2 , . . . , N } , if the algorithm pulls some arm a ∈ [ K ] / { 1 } at iteration N and doesn't pull a for the next R iterations, then,

where for every alternative arm j ∈ [ K ] / { 1 } ,

is the modified empirical index of that arm at iteration N

Proof. For cleaner presentation, we use b to denote b ( N,R ) . We also use ˜ µ j , ˜ x 1 ,j , ∆ ˜ N j and ˜ N j , respectively, to denote ˜ µ j ( N ) , ˜ x 1 ,j ( N ) , ∆ ˜ N j ( N,R ) and ˜ N j ( N ) for every arm j ∈ [ K ] , whenever it doesn't cause confusion.

AT2 Algorithm: Let

denote the the difference between the empirical indexes of the two arms a and b at iteration N . Note that S a,b ( ˜ N , ˜ µ ) depends only on the tuple of variables ( ˜ N 1 , ˜ N a , ˜ N b , ˜ µ 1 , ˜ µ a , ˜ µ b ) . In the following argument, we apply mean value theorem over S a,b and the empirical indexes I a and I b , treating them as functions of ˜ N 1 , ˜ N a , ˜ N b , ˜ µ 1 , ˜ µ a , and ˜ µ b .

Using the multivariate mean value theorem, we have,

where ∆ ˜ µ j = ˜ µ j ( N + R ) -˜ µ j ( N ) for j = 1 , a, b , and ( ˆ N , ˆ µ ) = ( ˆ N 1 , ˆ N a , ˆ N b , ˆ µ 1 , ˆ µ a , ˆ µ b ) such that, for j = 1 , a, b , ˆ µ j lies between ˜ µ j ( N ) and ˜ µ j ( N + R ) , and ˆ N j lies in [ ˜ N j ( N ) , ˜ N j ( N + R ) ] . Note that there is no contribution on the RHS in (83) due to ˜ N a , because ˜ N a ( · ) doesn't change during

iterations N +1 , N +2 , . . . , N + R , owing to our assumption that the algorithm doesn't pull arm a during the mentioned iterations.

First we consider the partial derivatives of S j,b with respect to µ 1 , µ j and µ b in (83). We have

where ˆ x 1 ,j = ˆ N 1 ˆ µ 1 + ˆ N j ˆ µ j ˆ N 1 + ˆ N j for j = a, b .

Since R ≤ N , we have for j = 1 , a, b , ˆ N j ≤ ˜ N j + N = Θ ( N ) (by Lemma G.13). As a result, using (7) of Appendix B, both d 1 (ˆ µ 1 , ˆ x 1 ,a ) and d 1 (ˆ µ 1 , ˆ x 1 ,b ) are Θ (1) . Using this, ∣ ∣ ∣ ∂S a,b ∂µ 1 ( ˆ N , ˆ µ ) ∣ ∣ ∣ is O ( N ) .

Similarly,

Using the same argument as for the partial derivative of S a,b with respect to µ 1 , we have ∣ ∣ ∣ ∂S a,b ∂µ j ( ˆ N , ˆ µ ) ∣ ∣ ∣ = O ( N ) for both j = a, b .

Therefore, the contribution in the RHS of (83) due to the noisiness in the empirical means ˜ µ 1 , ˜ µ a , ˜ µ b is bounded above by,

where (1) follows since for N ≥ T 0 and j = 1 , a, b , | ∆ µ j | = O ( N -3 α/ 8 ) .

˜

Now considering the partial derivative of S a,b ( · ) with respect to N 1 , we get

Using Lemma G.13, ˜ N j ( N ) and ˜ N j ( N + R ) are both Θ ( N ) for all j = 1 , a, b and N ≥ T 6 , since R ≤ N . As a result, using (7), (8), we have,

for every j, k ∈ { 1 , a, b } , and tuple ( N ′ , µ ′ ) = ( N ′ 1 , N ′ a , N ′ b , µ ′ 1 , µ ′ a , µ ′ b ) having N ′ i ∈ [ ˜ N i ( N ) , ˜ N i ( N + R ) ] and µ ′ i lying between ˆ µ i and ˜ µ i , for every i = 1 , a, b .

Therefore, applying the mean value theorem, we get,

Similarly, considering the partial derivative with respect to N b ,

Now, using all these upper bounds in the RHS of (83), we obtain

Since arm a was pulled in iteration N , we have I a ( N -1) ≤ I b ( N -1) . Also since arm b was not pulled in iteration N , its empirical index remains unchanged, i.e. I b ( N -1) = I b ( N ) . Combining these two observations, we have I a ( N -1) ≤ I b ( N ) .

As a result,

where the last step follows from the fact that, the partial derivatives of I a ( · ) with respect to N a is d ( µ a , x 1 ,a ( N )) ≤ d ( µ a , µ 1 ) . As a result, since arm a was pulled in iteration N , the increment I a ( N ) -I a ( N -1) in I a ( · ) is upper bounded by d ( µ a , µ 1 ) .

Therefore (85) implies,

Now we consider two possibilities:

Case I d ( ˜ µ 1 , ˜ x 1 ,a ) -d ( ˜ µ 1 , ˜ x 1 ,b ) < 0 : In this case, by (85), S a,b ( ˜ N ( N + R ) , ˜ µ ( N + R )) is upper bounded by,

By Lemma G.13 and (6), since both ˜ N 1 ( N ) and ˜ N b ( N ) are Θ ( N ) , we have d ( ˜ µ b , ˜ x 1 ,b ) = Θ (1) . As a result, (81) follows.

Case II d ( ˜ µ 1 , ˜ x 1 ,a ) -d ( ˜ µ 1 , ˜ x 1 ,b ) ≥ 0 : In this case, the RHS of (85) can be rewritten as,

Using (76) in Lemma G.14, the upper bound becomes

By (86), we have,

Expanding the empirical index terms, we get,

By Lemma G.13 we have ˜ N b = Θ ( N ) . As a result, upon dividing both sides of the above inequality by ˜ N b and after some re-arrangement of terms, we obtain,

Using the above inequality in (89), we get the upper bound,

By Lemma G.13 and (6), we have, ˜ N j ˜ N b d ( ˜ µ j , ˜ x 1 ,j ) = Θ (1) . Therefore (81) follows.

IAT2 Algorithm: The proof of (82) for IAT2 is very similar to the proof of (81) for AT2. We first consider the difference of the modified empirical indexes between the two arms a and b ,

Following a similar procedure as the AT2 algorithm, we apply the mean value theorem to obtain,

where ∆ ˜ µ j = ˜ µ j ( N + R ) -˜ µ j ( N ) , and ( ˆ N , ˆ µ ) = ( ˆ N 1 , ˆ N a , ˆ N b , ˆ µ 1 , ˆ µ a , ˆ µ b ) , such that, ˆ µ j lies between ˜ µ j ( N ) and ˜ µ j ( N + R ) , and ˆ N j ∈ [ ˜ N j ( N ) , ˜ N j ( N + R ) ] , for every j = 1 , a, b .

The contribution to (92) due to noise in ˜ µ is O ( N 1 -3 α/ 8 ) by the same argument we proved (84) for AT2.

Now we consider the partial derivatives of S ( m ) a,b with respect to N 1 , N a , N b . Following the same steps as used for AT2 algorithm, we have,

As a result, the contribution to (92) due to ˜ N 1 and ˜ N b is,

Therefore, adding the contributions of the noise in ( ˜ µ j ) j =1 ,a,b , (92) can be further upper bounded by,

Now, we find an upper bound to S ( m ) a,b ( ˜ N ( N ) , ˜ µ ( N )) . Since the algorithm pulls arm a and doesn't pull arm b at iteration N , we must have

Therefore,

where (1) follows from the fact that ˜ N a ( N ) = Θ ( N ) for N ≥ T 6 , and (2) follows using the arguments used while bounding I a ( N ) -I a ( N -1) in (86).

Putting this in (93), we get,

Now there can be two cases.

Case I d ( ˜ µ 1 , ˜ x 1 ,a ) -d ( ˜ µ 1 , ˜ x 1 ,b ) < 0 : Then, using the same argument as was used for Case I of AT2 algorithm, we get (82).

Case II d ( ˜ µ 1 , ˜ x 1 ,a ) -d ( ˜ µ 1 , ˜ x 1 ,b ) ≥ 0 : Using (76) of Lemma G.12, the RHS in (95) is upper bounded by,

Using (94), we have which implies,

where (1) follows from the fact that ˜ N a ( N ) and ˜ N b ( N ) are both Θ ( N ) , causing log ( ˜ N a ( N ) ˜ N b ( N ) ) = O (1) for N ≥ T 6 . Note that the above inequality is same as (90) obtained in Case II of AT2. Now (82) follows using exactly the same argument as in the Case II of AT2 after (90).

For every a ∈ [ K ] / { 1 } , N ≥ T 6 and R ∈ { 1 , 2 , . . . , N } , we have,

where: (1) follows by Lemma G.3; (2) follows since R ≤ N ; and (3) follows from the fact that I a ( · ) can increment by atmost max { d ( µ 1 , µ a ) , d ( µ a , µ 1 ) } in one iteration.

Since, at every iteration N ≥ 1 and for every arm, the modified empirical index differ from the empirical index by atmost log( N ) , we have,

where (i) follows because R ≤ N .

Using (96) and (97), the following corollary follows from Lemma G.16,

Corollary G.2. For AT2 (1) and IAT2 (2) algorithms, for N ≥ T 6 and R ∈ { 1 , 2 , . . . , N } , if the algorithm pulls some arm a ∈ [ K ] / { 1 } at iteration N and doesn't pull a for the next R iterations, then,

where C 3 , C 5 > 0 are constants independent of the sample paths.

Proof of Lemma G.2: We prove the proposition for the AT2 algorithm. The proof for IAT2 algorithm follows the exact same argument by replacing I with I ( m ) .

In the proof, we argue via contradiction. We show that, there exists constants M 5 ≥ 1 and C 1 > 0 independent of the sample paths, such that for N ≥ max { M 5 , T 6 } , if the algorithm pulls some arm a ∈ [ K ] / { 1 } at iteration N and doesn't pull it for the next R ( N ) def. = ⌈ C 1 N 1 -3 α/ 8 ⌉ iterations, then at iteration τ ( N,R ( N )) , the algorithm pulls arm b ( N,R ( N )) , even though arm a has empirical index strictly less than that of arm b ( N,R ( N )) .

Using Corollary G.2, there exists constants C 3 , C 5 , C 6 > 0 independent of the sample paths, such that, for N ≥ T 6 and R ∈ { 1 , 2 , . . . , N } ,

We define

where D 1 and D 2 are the constants introduced in Lemma G.15. Let M 52 ≥ 1 to be the smallest number such that, every N ≥ M 52 satisfies ⌈ C 1 N 1 -3 α/ 8 ⌉ < N . Since C 1 > D 1 , by the definition of M 51 in the proof of Lemma G.15, we have M 52 > M 51 .

Now let R ( N ) = ⌈ C 1 N 1 -3 α/ 8 ⌉ . Then by Lemma G.15, for N ≥ max { M 52 , T 6 } , since R ( N ) > ⌈ D 1 N 1 -3 α/ 8 ⌉ , we have,

Since arm b ( N,R ) is not pulled between the iterations τ ( N,R ) and N + R , we have

for all t ∈ { τ ( N,R ) -N, τ ( N,R ) -N +1 , . . . , R } .

As a result, by definition of b ( N,R ) we have,

In the rest of the proof, we denote b ( N,R ( N )) and τ ( N,R ( N )) -N , respectively, using b ( N ) and τ b ( N ) .

Therefore, using Corollary G.2, we have, for N ≥ max { M 52 , T 6 } ,

where: (1) follows using (101) and τ b ( N ) ≤ R ( N ) , (2) follows since R ( N ) ≤ ( C 1 +1) N -3 α/ 8 , and (3) follows by letting C 6 = C 5 ( C 1 +1)( C 1 +2) .

We now take M 53 ≥ 1 to be large enough, such that, C 6 M -3 α/ 8 53 < C 4 . Let M 5 = max { M 52 , M 53 } . Then (102) implies, for N ≥ max { M 5 , T 6 } and R ( N ) = ⌈ C 1 N 1 -3 α/ 8 ⌉ , if the algorithm picks some arm a ∈ [ K ] / { 1 } at iteration N and doesn't pick a for the next R ( N ) iterations, then, at iteration N + τ b ( N ) , the algorithm picks arm b ( N ) , even though, I a ( N + τ b ( N ) -1) -I b ( N ) ( N + τ b ( N ) -1) < 0 . Thus we arrive at a contradiction.

H Proof of Theorem 3.1

By Proposition 3.1, we know that, for every a ∈ [ K ] and N ≥ T stable ,

Recall the functions W a ( · , · ) defined in Appendix G.2.2. Note that for every allocation ω = ( ω a ) a ∈ [ K ] , and a ∈ [ K ] / { 1 } , we have,

As a result, for every a ∈ [ K ] / { 1 } , the partial derivatives of W a ( ω 1 , ω a ) with respect to ω 1 and ω a are both O (1) . Therefore, using the mean value theorem, for every a ∈ [ K ] / { 1 } and N ≥ T stable , we have,

Define the normalized index of every arm as

Also by (54), W a ( ω ⋆ 1 , ω ⋆ a ) = I ⋆ = T ⋆ ( µ ) -1 for every alternative arm a ∈ [ K ] / { 1 } .

Therefore (103) gives us,

Combining (104) and (105), we get

for every a ∈ [ K ] / { 1 } . Hence, we can find a constant C 2 > 0 , independent of the sample paths, such that,

for every N ≥ T stable and a ∈ [ K ] / { 1 } . As a result, for N ≥ T stable , we have,

The threshold function β ( N,δ ) deciding our stopping condition satisfies,

for constants C 3 , C 4 , C 5 > 0 . For every δ > 0 , we define the deterministic quantity,

Now we make the following observations about t max ,δ ,

Note that N T ⋆ ( µ ) -C 2 N 1 -3 α/ 8 increases linearly in N and β ( N,δ ) is O (log log( N ) + log(1 /δ )) , for a fixed δ > 0 . Hence, t max ,δ is finite for every δ > 0 .

We have β ( N,δ ) ≥ log(1 /δ ) and N T ⋆ ( µ ) -C 2 N 1 -3 α/ 8 < N T ⋆ ( µ ) . As a result, t max ,δ is atleast the iteration at which N T ⋆ ( µ ) exceeds log(1 /δ ) , which is atleast T ⋆ ( µ ) log(1 /δ ) . This implies t max ,δ ≥ T ⋆ ( µ ) log(1 /δ ) . As a result, t max ,δ →∞ as δ → 0 .

If τ δ > T stable , then min a ∈ [ K ] / { 1 } I a ( N ) exceed β ( N,δ ) before the lower bound of min a ∈ [ K ] / { 1 } I a ( N ) in (106) exceeds β ( N,δ ) . As a result, τ δ ≤ t max ,δ , whenever τ δ ≥ T stable . This gives us the upper bound,

We have E µ [ T stable ] < ∞ , which also implies T stable < ∞ a.s. in P µ . Now using (108), we get,

Similarly, we have,

Therefore to prove asymptotic optimality, it is sufficient to prove lim sup δ → 0 t max ,δ log(1 /δ ) ≤ T ⋆ ( µ ) .

Let s max ,δ = t max ,δ -1 . Note that lim sup δ → 0 t max ,δ log(1 /δ ) = limsup δ → 0 s max ,δ log(1 /δ ) . By definition of t max ,δ , we have

After some rearrangement of terms, upon dividing both sides by log(1 /δ ) , we get,

By Observation 2, s max ,δ = t max ,δ -1 →∞ as δ → 0 . As a result, the above inequality implies,

We already argued that lim sup δ → 0 t max ,δ log(1 /δ ) ≤ T ⋆ ( µ ) . As a result, we have a constant C 6 > 0 , such that t max ,δ ≤ C 6 log(1 /δ ) for all δ . By (108) we have:

which implies,

Putting t max ,δ ≤ C 6 log(1 /δ ) in the above inequality,

Since τ δ ≤ max { T stable , 1 + t max ,δ } , we can find a constant C > 0 such that τ δ ≤ max { T stable , T ⋆ ( µ ) log(1 /δ ) + C (log(1 /δ )) 1 -3 α/ 8 } . Hence Theorem 3.1 stands proved.

I Extending the proposed algorithm to distributions with bounded support

Wedescribe a natural extension of AT2 and IAT2 algorithms to bandit instances from a non-parametric family. We conduct experiments to compare the proposed algorithm with the existing ones. We consider the class of distributions having their supports contained in [0 , 1] , which we denote by F [0 , 1] . This is similar to the assumptions made in [16]. Some definitions are in order. We use µ ( G ) to denote the mean of distribution G ∈ F [0 , 1] .

For every F ∈ F [0 , 1] and x ∈ [0 , 1] , we define KL + and KL -

At iteration N of the algorithm, let ˜ F a ( N ) be the empirical distribution of the samples collected from some arm a ∈ [ K ] , ˜ F ( N ) = ( ˜ F a ( N ) : a ∈ [ K ]) , ˜ N a ( N ) be the total no. of samples collected from a till N , and ˜ N ( N ) = ( ˜ N a ( N ) : a ∈ [ K ]) . Let ˜ µ a ( N ) = µ ( ˜ F a ( N )) and ˆ i N = arg max a ∈ [ K ] ˜ µ a ( N ) . We now define:

̸

At every iteration, we compute the empirical index of every arm a = ˆ i N as:

̸

and the anchor function as,

The AT2 and IAT2 algorithm for the class F [0 , 1] , respectively, follows the same steps as in (1) and (2) with the anchor and index functions defined as above.

We experimentally demonstrate the performance of the proposed algorithms in Appendix J.5.

J Experiments

J.1 Dynamics of the algorithms

Experiment 1 (Gaussian and Bernoulli bandits with well-separated arms): In the main text (Figure 1), we presented the evolution of normalized indexes for the sub-optimal arms for AT2, when run without the stopping rule. Numerically, we observe similar plots for normalized indexes even for the other algorithms: 0 . 5 -EB-TCB (proposed in [16] with β = 0 . 5 ), and TCB (proposed in [22]). Hence, we do not report them. However, we do observe differences in the evolution of the anchor function value across these algorithms. We present this in two different settings in the current section.

Interestingly, as per our implementations, we observe that only AT2 satisfies the asymptotic optimality conditions, maintaining the anchor function close to 0 , in addition to maintaining the equality of the normalized indexes.

In this section, we consider the following two examples:

Gaussian bandit. In the first setup (Figure 4), we consider a 4 armed Gaussian bandit with unit variance and mean vector µ = [10 , 8 , 7 , 6 . 5] . This is the same setting as in Section 6 from the main text.

Bernoulli bandit. In the second setup (Figure 5), we consider a Bernoulli bandit with means µ = [0 . 91 , 0 . 73 , 0 . 64 , 0 . 59] .

In Figures 4 and 5, we plot the evolution of anchor function value for the three algorithms in the two settings, without implementing the stopping rule. The solid lines in the figure represent the average of anchor function over 4 , 000 independent runs. The shaded bands around the sold lines (almost invisible in these figures), represent 2 standard deviation bands around the mean.

Weobserve that only AT2 maintains the anchor function value close to 0 . Our experiments suggest that that TCB algorithm, as per our implementation, doesn't satisfy the asymptotic optimality conditions.

J.2 Sample complexity comparison

In Section 6 in the main text, we compared the sample complexities (SC) of the three algorithms on a well-separated Gaussian bandit (Figure 2). In this section, we compare the SC of all the algorithms, as a function of different parameters. We consider harder Gaussian as well as Bernoulli bandit instances (with means close to each other), presented below.

Gaussian bandit. A 4 armed Gaussian bandit with unit variance and mean vector µ = [7 . 25 , 7 . 05 , 7 , 7 . 1] so that the means are closer together.

Bernoulli bandit. As a second example, we consider a 4 -armed Bernoulli bandit with close-by means: µ = [0 . 99 , 0 . 96 , 0 . 95 , 0 . 97] .

Experiment 2 (SC as function of β ): In this experiment, we compare SC of (I)AT2, (I)TCB, β -EB(I)TCB algorithms, for β ∈ [0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8] , on the Gaussian instance (Figure 6) and the Bernoulli instance (Figure 7), described above. The error probability δ in both these experiments is set to 0 . 001 . All the algorithms use the same forced exploration rule and stopping rules.

The lines with the markers in the figures represent the average number of samples generated before stopping, averaged over 4 , 000 independent simulations, while the shaded regions denote 2 standard deviations around the mean. We also report the average sample complexity and the standard deviation of the average sample complexity for AT2, IAT2, TCB, and ITCB, across 4 , 000 independent simulations.

In both these simulations, we observe that AT2 and IAT2, respectively, have about 5% lower sample complexity compared to TCB and ITCB.

Algorithm

Avg. SC

AT2

2013.0

17.85

IAT2

1925.9

16.36

0

.

5

-EB-TCB

2084.84

17.74

0

.

5

-EB-TCBI

1987.92

16.33

TCB

2109.82

17.82

ITCB

2041.03

16.55

Algorithm

Avg. SC

AT2

4735.09

50.4

IAT2

4412.16

27.36

0

.

5

-EB-TCB

4735.65

45.44

0

.

5

-EB-TCBI

4426.59

27.27

TCB

4972.07

49.64

ITCB

4681.63

28.52

Experiment 3 (SC as function of δ ): In Figure 8 and Figure 9, we plot the sample complexities of the three algorithms - AT2, 0 . 5 -EB-TCB, and TCB - as a function of δ , for the Gaussian and Bernoulli bandits considered in Experiment 2 above. All the algorithms use the same forced exploration and stopping rules. We observe that AT2 consistently outperforms both the previously known algorithms, and the gap in performance increases as we reduce δ .

Experiment 4 (SC as a function of number of arms). We plot the number of samples needed by the three algorithms, as a function of number of arms in the bandit instance. δ is set to 0 . 001 in this experiment.

For scalability, in this experiment, we consider a simple Gaussian bandit (well-separated means) with all arms having a unit variance. Arm 1 is optimal with mean 10 . To study the effect of number of arms on sample complexity, we choose all the other arms to be same with mean 8 . Thus, the bandit instances have Gaussian arms with unit variance, and means

As in the earlier experiments, for fair comparison, all the algorithms are implemented with the same forced exploration and stopping rules. Results are presented in Figure 10. We observe that the sample complexity increases linearly with number of arms for the three algorithms. In this experiment, the performance of TCB and AT2 looks comparable, with TCB requiring slightly more number of samples. However, the gap in their performance is expected to increase for smaller values of δ .

J.3 Runtime comparison

Experiment 5: In this experiment, we compare the run-time of (I)AT2 and (I)TCB algorithms on a 4 armed Gaussian bandit with means µ = [10 , 9 . 4 , 7 , 6 . 5] and unit variance, averaged over longer

100,000 simulations. δ is set to 0 . 001 , and the four algorithms use the same forced exploration and stopping rules.

J.4 Effect of forced-exploration parameter α on sample complexity

In this section, we provide a numerical evaluation of the impact of forced exploration on the performance of AT2 and IAT2. Our experiments suggest that unlike IAT2, AT2 needs forced exploration. On instances where the second and third best arms have equal means, AT2 might see some bad samples from the best arm in the beginning. As a result, without forced exploration, it will sample the second and the third best arms forever. However, we see that AT2 performs sufficient exploration for instances having all arms with different means. Note that similar observations were made in [16] for β -EB-TCB.

Experiment 6: To see the above mentioned behavior of AT2, we study the performance of AT2 and IAT2 on the following two bandit instances.

A 4 armed Gaussian bandit with unit variance and mean vector µ = [7 . 25 , 7 . 05 , 7 , 7 . 1] , so that the means are close together, yet all different.

A 4 armed Gaussian bandit with unit variance and mean vector µ = [7 . 25 , 7 , 7 , 7] , so that the three suboptimal arms have equal means.

Intuitively, it might appear that forced exploration could significantly increase the sample complexity of AT2 and IAT2. However, in Figure 11, we see that IAT2's performance remains unaffected and AT2 performs at least as well as IAT2 with moderate amount of forced exploration on the first instance, where the gap between the means is small.

Figure 12 shows that without forced exploration, AT2's sample complexity blows up on the instance with equal sub-optimal arm means. We see that on this instance, IAT2 performs significantly better than AT2. Furthermore, IAT2's sample complexity remains almost unaffected with respect to the amount of forced exploration done.

J.5 Experiments for bandits with bounded-support distributions

In this section, we experimentally demonstrate the performance of a natural extension of AT2 to a non-parametric setting of bandits with arms having distributions supported in [0 , 1] (see Appendix I for the modified AT2). This is the setting considered in, for example, [16].

Experiment 7: Consider a 4 -armed bandit with the following arm distributions:

Here, the arms have means

While we do not provide the analysis of the algorithm for this setting, numerically we observe that even in this non-parametric setting, extension of AT2 to this setting outperforms β -EB-TCB, and a corresponding natural extension of TCB to this setting.

Reproducibility: Our code is implemented in Julia 1.7.1 , and the plots are generated with the Plots.jl package. Other dependencies are listed in the Readme.md file, which also includes instructions to reproduce the figures and tables presented here. We build upon the publicly available code for [16]. Our experiments are conducted on an institutional cluster computing facility having an Intel Xeon Gold 6130 2.1GHz CPU with 32 cores.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The optimality of the proposed algorithm is proven in Theorem 3.1. We also demonstrate the dynamics of the algorithm, as proposed in Theorem 4.1 via numerical simulations.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations of the results presented in this paper due to model assumptions such that independence, considering bandit instances from an SPEF, etc. at the end of Section 1.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We include all the assumptions made in the statements of the results. We provide all the formal proofs in Appendix, and refer to them. All the theorems/lemmas/propositions are numbered and referenced appropriately.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our key contribution is a new algorithm with its theoretical guarantees. We perform numerical simulations to support the theoretical results. We include all the implementation details with the choices for different parameters in our numerical experiments discussion in the Appendix J.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include the code with scripts for reproducing the numerical results.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We include the choices of all the parameters in each simulation.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We include mean and standard deviations in all our experiments, along with the assumptions made (independence of different runs of same simulation).

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We use an institutional computing facility for our experiments (see, end of Appendix J).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The results of this work are theoretical in nature. We appropriately reference all existing works to the best of our knowledge. We include justifications for all the results, and implementation details for reproducing the simulation results of this work.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The main contributions of this work are theoretical in nature. While we propose a new algorithm, there are no harmful societal impacts or consequences of this research.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We run numerical simulations. We do not use any publicly available datasets for our experiments.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Please see paragraph on Reproducibility in Appendix J.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release any new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowd sourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the top-2 algorithm for identifying the best arm in multi-armed bandits. The authors introduce a new approach for determining the optimal $\\beta$ for sampling the empirical best and best challenger arms. This novel method relies on a function of allocations anchored at a threshold, which is simple in concept.\n\nSTRENGTHS:\n1. The paper introduces a novel method for determining the optimal $\\beta$ in Top-2 algorithms, which has been a popular problem in the field. Empirical results demonstrate that the proposed method offers improved sample complexity and computational efficiency compared to existing approaches.\n2. The analysis leverages fluid dynamics and the implicit function theorem to establish the algorithm's optimality, marking a unique and intriguing contribution to the field of bandit problems.\n\nWEAKNESSES:\n1. One weakness of the paper is the use of the forced exploration stage in the AT2 algorithm (Line 1-2), which ensures that all the arm means are estimated with sufficient accuracy. It is not clear whether this is crucial for the effectiveness of the algorithm or simply for the ease of theoretical proof. Indeed, forced exploration might harm the empirical performance of BAI but significantly simplifies the asymptotic theoretical analysis.\n2. The second weakness of the paper is that all the analyses are asymptotic, although I understand that achieving non-asymptotic results might be challenging for the top-2 algorithm.\n\nQUESTIONS:\n1. Line 634-637: What about a Bernoulli distribution with mean zero?\n2. Line 144: Another widely used definition for a $\\delta$-correct algorithm is $\\mathbb{P}\\left(\\tau_\\delta<\\infty\\right) = 1$ and $\\mathbb{P}\\left( k_{\\tau_\\delta} \\neq 1\\right) \\leq \\delta$. Are there any concerns with this definition?\n3. It is surprising that TCB does not satisfy the asymptotic optimality conditions. Have you verified the correctness of the implementation? Additionally, did you use the same forced exploration rule and stopping rule for all algorithms in the numerical results? Finally, I suggest including a theoretical analysis of the computational complexity for the proposed algorithm.\n\nLIMITATIONS:\nThe main contributions of this work are theoretical in nature. I don't foresee any direct negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the problem of fixed-confidence BAI in stochastic multi-armed bandits. Simply stated, the problem entails finding the best arm (the arm with the largest mean reward) as quickly as possible, subject to an upper bound on the error probability. There exist a plethora fo works that investigate this problem, derive problem instance-dependent lower bounds on the asymptotic growth rate of the expected stopping time, and devise algorithms that achieve the asymptotic lower bound; here, the asymptotics is one of vanishing error probabilities. One class of efficiently implementable algorithms for this problem that has gained significant attention in the recent years is that of top-2 sampling algorithms. The basic premise on which such algorithms function is to pull the empirical best arm at any given time with a certain constant probability, say $\\beta$, and to pull the empirically second-best arm (or the challenger arm) with the remaining probability. Despite its elegance and ease of implementability, the aforementioned $\\beta$-top-2 sampling algorithm an algorithm falls short of achieving the instance-dependent lower bound. The key to achieving the lower bound, as it turns out, is to set $\\beta=\\beta^*$, where $\\beta^*$ depends on the underlying problem instance and is therefore unknown to the algorithm. While several attempts to learn the value of $\\beta^*$ as part of the algorithm have taken place in the recent past, the current work accomplishes the same objective elegantly, using the theory of ODEs. Notably, the paper proposes an algorithm for BAI, and analyses the fluid dynamics (ODE) corresponding to a continuous version of the same. Implicit function theorem is used to demonstrate the existence and uniqueness of the solutions to the ODE, and to further demonstrate that the arm allocations under the algorithm closely follows the trajectory coming from the ODE. It is then argued that the limiting solution of the ODE (and hence the asymptotic arm allocations under the algorithm) matches with the optimum allocations corresponding to the underlying instance, thereby proving the asymptotic optimality of the proposed algorithm.\n\nSTRENGTHS:\nThe use of the theory of ODEs and the implicit function theorem to demonstrate the asymptotic optimality of the proposed algorithm is the novel aspect of this paper. To the best of my knowledge, such an approach to the demonstration of asymptotic optimality does not exist in the literature on fixed-confidence BAI. Furthermore, while the concept of *anchor function* appearing in the paper has its roots in Garivier and Kaufmann's paper on BAI, a careful analysis of its behaviour using the theory of ODE to demonstrate asymptotic optimality of the proposed algorithm is impressive. \n\n(I must acknowledge, though, that I was unable to go through the extensive and detailed appendix due to paucity of time.)\n\nWEAKNESSES:\nThe writing of the paper can be slightly improved. While the authors have done a commendable job in explaining the intuition behind their detailed proofs, I have some minor suggestions that might possibly improve the readability of the paper.\n\n1. In the paragraphs leading up to Proposition 2.1, the authors introduce the notation $x\\_{1,a}=\\dfrac{\\omega\\_1 \\ x\\_1 + \\omega\\_a \\ x\\_a}{\\omega\\_1 + \\omega\\_a}$. Shortly thereafter, the authors use the phrase \"define $N\\_{1,1}$ as the value of $N\\_1$ at which $\\sum\\_{a \\neq 1} \\dfrac{d(\\mu\\_1, x\\_{1,a})}{d(\\mu\\_1, x\\_{1,a})}=1$ for a given allocation $\\boldsymbol{N}\\_{\\bar{B}^c}$\", without stating the (new) definition of $x\\_{1,a}$. The actual definition of $x\\_{1,a}$ only appears later, in the first part of Proposition 2.1. I suggest that the authors move the definition of $x\\_{1,a}$ to an earlier paragraph if possible.\n\nSome minor comments:\n\n2. \"alocation\" in line 160 --> \"allocation\".\n\n3. $f(\\boldsymbol{\\mu}, a, N)$ in line 291 --> $f(\\boldsymbol{\\mu}, a, \\boldsymbol{N})$.\n\nQUESTIONS:\nCan the authors explain what role the anchor function $g$ plays at a higher level. It seems almost unintuitive that this function, together with the empirical means and allocations, should dictate the sampling of the best empirical arm or challenger arm at any given time (items 3,4 in lines 208-211), in order to match with the optimal allocations in the long run. For a contextual comparison, in the TCB and ITCB algorithms of [21], the place of the anchor function is taken by the empirical problem complexity. Given that the empirical terms are expected to be close to their true values in the long run, the empirical problem complexity is likely to be close to the true complexity. It is then intuitive to expect that any sampling rule that is based on the empirical problem complexity, is likely to lead to sampling the arms according to the optimal allocation in the long run. \n\nAlong these/similar lines, can the authors provide a high-level intuition for why looking at the anchor function and its long-term behaviour is pivotally connected to learning the optimal $\\beta$ and thereby matching the arm selection frequencies with the optimal allocation?\n\nLIMITATIONS:\nNA.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-04"}

comments: SUMMARY:
This paper considers the problem of identifying an optimal "top-2" algorithm for the best arm identification problem in the bandits framework. The question of how to allocate pulls between incumbents and challengers is considered. An algorithm for achieving the best balance is described.

STRENGTHS:
"Top-2" approaches for best arm identification are powerful and increasingly popular, so this paper's focus is an important area of study. Posing the problem in terms of fluid dynamics could potentially be a powerful way of modeling the problem.

WEAKNESSES:
I found the paper to be generally very hard to follow. The language is awkward in many places. I do not find that the introduction makes it clear how the paper's contributions are achieved. Explanations are very math-heavy and hard to follow and I find the theoretical results (e.g., Proposition 2.1) are not understandable. The paper also references many concepts without explaining them well (e.g., IFT, GLLR, ITCB). 

The presentation of this paper needs to be significantly improved before it is ready for publication.

QUESTIONS:
If we were to pull both of the top-2 arms instead of randomizing between them, how different would the sample complexity be?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on best arm identification (BAI) under the fixed confidence setting. It assumes the underlying distributions are from the single parameter exponential function. It discusses the problem of how to find the optimal $\\beta$ for Top-Two type of algorithm for BAI, where $\\beta$ denotes the probability of pulling the empirical best arm at each step. It proposes two algorithms, called Anchored Top-2 (AT2) algorithm and an improved version called Improved AT2 (IAT2) algorithm. It shows that both of the algorithms reaches the optimal sample complexity when $\\delta \\to 0$. A key idea the authors propose is to identift the underlying fluid dynamics using the ODEs tracked by the proposed algorithms. This idea helps to prove the convergence of the algorithms. It also does simulations to show the index values for each arm at different time steps for AT2 and compares the sample complexity of AT2 with other popular Top-Two algorithm.\n\nSTRENGTHS:\n1. The paper discusses an important question which is to find the optimal $\\beta$ in the Top-Two algorithm and shows a sound solution (AT2 algorithm) to solve this problem.\n2. The contributions are solid and they novelly use the fluid dynamics to describe the behaviors of the proposed algorithms.\n3. The tools for proving the upper bound of the sample complexity of the proposed algorithms are also novel enough.\n\nWEAKNESSES:\n1. The paper is rather technical and lacks some intuitive explanations of different parts. It would be nice to include a picture to desribe the actual behavior of the paths captured by the ODEs if possible. Or just provide more intuitions for the steps of the algorithms.\n2. The experiments lack some more explanations to discuss the meanings behind.\n\nQUESTIONS:\n1. possible typos: \"There\" -> \"Their\" in line 88?\n2. What is $a$ in the equation at the end of page 3?\n3. Why the threshold for function $g(\\cdot)$ in the algorithm is 0? Is it possible or trivial to change to other thresholds?\n4. Why the stopping condition is ignored as mention in line 249 for the fluid dynamics?\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is about fixed confidence bandit best arm identification. The authors study the top-two family of algorithms. Those algorithms identify at each time step a leader and a challenger arm, and sample one of those two arms: typical top-two algorithms in the literature sample the leader with a fixed probability $\\beta$ chosen in advance. A top-two algorithm that uses a fixed $\\beta$ cannot have optimal expected sample complexity for all problems, since the optimal proportion of samples for the best arm might not be $\\beta$. In order to derive an asymptotically optimal top-two algorithm, the authors introduce an adaptative choice of $\\beta$, and study its dynamic through the study of a continuous time fluid dynamic.\n\nSTRENGTHS:\n- The method proposed is the first fully adaptive choice of the proportion $\\beta$ that ensures asymptotic optimality for all single parameter asymptotic families. Another optimal method was already proposed in [28], but only for Gaussian distributions.\n\n- The fluid dynamic model is particularly interesting: it gives a good intuition of why the dynamic for $\\beta$ converges to the right proportions. It is also a very original approach to the study of bandit identification algorithms.\n\n- The authors propose a full algorithm (leader and challenger choice as well as the new adaptive proportions $\\beta$), but the new adaptive proportions could presumably be used with other choices for the leader and challenger, and could become a generic tool for asymptotically optimal top-two algorithms.\n\n- The paper is clear and explains the idea behing the new method very well.\n\nWEAKNESSES:\n- The analysis is very asymptotic. It uses that after a while, the forced exploration of steps 1 and 2 in the algorithm ensures that all means are well estimated. That implicitly requires that $\\delta$ should be small enough that the stopping condition is not attained before that time. Most of the literature on top-two algorithms is also asymptotic so that is not very surprising, but there are also non-asymptotic results (for fixed $\\beta$): see [Jourdan & Degenne, Non-asymptotic analysis of a ucb-based top two algorithm].\n\n- the algorithm AT2 uses forced exploration, which could hurt in practice: if the parameter $\\alpha$ is set too high, the algorithm will explore some bad arms too much. On the other hand if it is set too low, the algorithm can get stuck sampling sub-optimal arms. Indeed, if $\\alpha$ is zero the algorithm is close EB-TC from [16] which can have horrible practical performance, and the new algorithms will likely get stuck in the same way: if there is one good arm and two bad arms with equal means but the good arm gets low empirical mean at the beginning, the algorithm will sample the two bad arms forever (see [16]). IAT2 should not suffer from the same issue, thanks to its added exploration mechanism.\nI did not find the value of $\\alpha$ used in the experiments and the effect of that parameter was not studied experimentally.\n\n- The analysis is very involved and might be hard to reuse or extend in other contexts. In particular, it uses a lot the particular shape of the optimality conditions for the sampling proportions. It is not clear if the right properties to use the same kind of technique will hold in other settings like structured bandits. This is more an open problem than a weakness of the paper.\n\nRemarks and typos:\n\n- A remark about this review: the paper including the appendix is 75 pages long, hence I could not check every proof.\n\n- An algorithm $\\beta$-EB-TCB is mentionned repeatedly, supposedly coming from [16]. There is no algorihtm called $\\beta$-EB-TCB in [16]. Is it $\\beta$-EB-TC? Likewise, $\\beta$-EB-ITCB does not exist in that paper: is it $\\beta$-EB-TCI?\n\nQUESTIONS:\n- The algorithm uses forced exploration to ensure that every arm is sampled some minimal amount (steps 1 and 2, lines 206 and 207). Suppose that instead, we use some variant of top-two without forced exploration, with the adaptive $\\beta$ that you propose, and prove that it naturally explores (we prove a \"sufficient exploration\" lemma as in [16]]). That is not possible with AT2 without forced exploration unless all arms have different means since it uses greedy leader and challenger, but it could be possible with IAT2. Would the analysis still apply? Or is the forced exploration also used for something else in the analysis?\n\n- The upper bound on the time needed to reach proportions close to the optimal proportions depends on the inverse of the minimal optimal proportion over the arms (line 262, line 321). That upper bound can be very large if the minimum is close to 0, which can happen if there is a very bad arm. Can this be avoided?\n\nLIMITATIONS:\nThe limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}