Title: Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates
Venue: NeurIPS 2024
Accepted: yes
Keywords: stochastic gradient bandit

arbitrary stepsize

global convergence
Abstract: We provide a new understanding of the stochastic gradient bandit algorithm by showing that it converges to a globally optimal policy almost surely using \emph{any} constant learning rate. This result demonstrates that the stochastic gradient algorithm continues to balance exploration and exploitation appropriately even in scenarios where standard smoothness and noise control assumptions break down. The proofs are based on novel findings about action sampling rates and the relationship between cumulative progress and noise, and extend the current understanding of how simple stochastic gradient methods behave in bandit settings.

Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates

Abstract

We provide a new understanding of the stochastic gradient bandit algorithm by showing that it converges to a globally optimal policy almost surely using any constant learning rate. This result demonstrates that the stochastic gradient algorithm continues to balance exploration and exploitation appropriately even in scenarios where standard smoothness and noise control assumptions break down. The proofs are based on novel findings about action sampling rates and the relationship between cumulative progress and noise, and extend the current understanding of how simple stochastic gradient methods behave in bandit settings.

1 Introduction

The stochastic gradient method has been ubiquitous in the field of machine learning for decades [4]. When applied to reinforcement learning (RL), a representative instantiation of stochastic gradient is the well known policy gradient [32] (or REINFORCE [34]) algorithm, where in each iteration an online sample is gathered using the current policy, from which a gradient estimate is obtained to conduct parameter updates. In the simplest setting of a stochastic bandit problem [16], where decisions matter only for one step, the REINFORCE policy gradient method becomes equivalent to the stochastic gradient bandit algorithm [31, Section 2.8]. Compared to other statistical methods, such as the upper confidence bound algorithm (UCB, [14, 3]), and Thompson sampling (TS, [33, 2]), the stochastic gradient bandit algorithm is conceptually simpler and more computationally efficient, as it does not calculate exploration bonuses nor posterior distributions. Moreover, the stochastic gradient method is highly scalable and naturally applicable to large scale neural networks [29, 30].

However, unlike UCB or TS, the stochastic gradient bandit algorithm does not have an equivalently well established and comprehensive theoretical footing. Given its pervasive success and widespread application in RL [30] and fine-tuning for large language models [26, 27]), it remains an important question to understand the success of stochastic gradient based algorithms in bandit-like settings, not only to bridge the gap between theory and practice, but also to identify more effective and robust variants. In this paper, we make a significant contribution to the theoretical understanding of the stochastic gradient bandit algorithm, bringing its justification closer to that of other less scalable but theoretically well established methods. In particular, we establish the surprising result that:

For any constant learning rate η > 0 , the stochastic gradient bandit algorithm is guaranteed to converge to the globally optimal policy almost surely.

Since learning rate is the only tuning parameter in the stochastic gradient bandit algorithm, this result offers a remarkable robustness for the method, that it converges to a near optimal policy, irrespective of the value of this hyperparameter! Analysis of this algorithm is challenging because it requires techniques for simultaneously handling non-convex optimization, stochastic approximation, and the exploration-exploitation trade-off. Prior theoretical work on the stochastic gradient algorithm has primarily focused on non-convex optimization and stochastic approximation, but understanding the simultaneous effect on exploration has been largely lacking.

Recently, significant progress has been made in establishing global convergence results for policy gradient (PG) methods. For example, it has been shown that using exact gradients, Softmax PG converges to a globally optimal policy asymptotically as the number of iterations t goes to infinity [1]. Subsequent work has demonstrated that the asymptotic rate of convergence is O (1 /t ) [23], albeit with problem and initialization dependent constants [22, 17]. The rate and constant dependence in the true gradient setting have been improved via several techniques, including entropy regularization [23], normalization [21], and using natural gradient (mirror descent) [1, 6, 15].

Unfortunately, in the online stochastic setting, where the policy gradient has to be estimated using the current policy to collect samples, these accelerated methods all obtain worse asymptotic results than the standard Softmax PG [20], failing to converge to a global optimum without careful design choices [19]. Yet in the same setting, standard Softmax PG has been shown to succeed in its simplest form, provided only that a sufficiently small learning constant rate η ∈ Θ(1) is used [24].

For stochastic gradient based methods, decaying or sufficiently small learning rates are used by almost all current approaches, motivated by classical convergence analyses from stochastic optimization [28, 12, 39, 38, 9, 37, 36, 24, 8]. Stationary point convergence is guaranteed for learning rates sufficiently small with respect to the smoothness of the objective function, while also decaying to zero at a precise rate if noise in the gradient estimator persists. In addition to appropriate learning rate control, many other techniques have been developed to control the effects of gradient noise, including regularization [38, 9], variance reduction [37], and carefully considering growth conditions [36, 24].

The technical challenges we face in the current study can be understood in the following aspects: (1) Using an arbitrarily large constant learning rate for online stochastic gradient optimization immediately renders the smoothness and noise control techniques mentioned above inapplicable. (2) With any constant learning rate η > 0 , the question of whether oscillation or convergence will ultimately occur needs to be addressed before even considering whether any convergence is to a global optimum. This additional level of complexity arises because the optimization objective is not necessarily improved monotonically in expectation. Finally, (3) The gradient bandit algorithm does not use any exploration bonus, which means that new techniques are required to demonstrate that it adequately balances the exploration-exploitation trade-off.

In this paper, we resolve the above difficulties by uncovering intriguing exploration properties of stochastic gradient when using any constant learning rate. In particular, we establish the following.

(i) In the stochastic online setting, with probability 1 , the stochastic gradient bandit algorithm will not keep sampling any single action forever, implying that it will exhibit a minimal form of exploration without any further modification. This asymptotic event (as t →∞ ) happens with probability 1 and holds for any constant learning rate η > 0 .

(ii) This result can then be leveraged to show that, as a consequence, given any constant learning rate, the stochastic gradient bandit algorithms will converge to the globally optimal policy as t →∞ , with probability 1 . That is, the probability of sub-optimal actions decays to 0 , even though some of them are taken infinitely often asymptotically.

2 Setting and Background

We consider the stochastic multi-armed bandit problem [16], specified by K actions and a true mean reward vector r ∈ R K , where for each action a ∈ [ K ] := { 1 , 2 , . . . , K } ,

where R max > 0 is the reward range, µ is a finite measure over [ -R max , R max ] , and P a ( x ) ≥ 0 is the probability density function with respect to µ . We use R a to denote the reward distribution for

action a defined by the density P a and base measure µ . The goal is to find a policy π θ ∈ [0 , 1] K to achieve high expected reward,

where π θ is parameterized by θ ∈ R K .

The gradient bandit algorithm. A natural idea to optimize Eq. (2) is to use stochastic gradient ascent, which is shown in Algorithm 1 and known as the gradient bandit algorithm [31, Section 2.8]. In Algorithm 1, in each iteration t ≥ 1 , the probability of pulling arm a ∈ [ K ] is given as

where θ t ∈ R K is the parameter vector to be updated. The following proposition shows that Algorithm 1 is an instance of stochastic gradient ascent with an unbiased gradient estimator [31, 24].

Proposition 1 (Proposition 2.3 of [24]) . Algorithm 1 is equivalent to the following update,

̸

where E t [ d π ⊤ θ t ˆ r t dθ t ] = d π ⊤ θ t r dθ t , and E t [ · ] is defined with respect to randomness from on-policy sampling a t ∼ π θ t ( · ) and reward sampling R t ( a t ) ∼ P a t . The Jacobian of θ ↦→ π θ := softmax( θ ) is ( d π θ dθ ) ⊤ = diag ( π θ ) -π θ π ⊤ θ ∈ R K × K , and ˆ r t ( a ) := I { a t = a } π θ t ( a ) · R t ( a ) for all a ∈ [ K ] is the importance sampling (IS) estimator, and we set R t ( a ) = 0 for all a = a t .

Known results on the convergence of the gradient bandit algorithm. Since Eq. (2) corresponds to a smooth non-concave maximization problem over θ ∈ R K [23], using Algorithm 1 with decaying learning rates is sufficient to guarantee convergence to a stationary point [28, 25, 12, 39]. However, this is insufficient to ensure the globally optimal solution of Eq. (2) is reached, since there exist multiple stationary points. More recently, guarantees of convergence to a globally optimal policy have been developed for PG methods in the true gradient setting [1, 23], where the algorithm has access to exact mean rewards. These results were later extended to achieve global convergence guarantees (almost surely) in the stochastic setting [38, 9, 37, 36, 24, 18]. However, these extended results have required decaying or sufficiently small learning rates, motivated by exploiting smoothness and combating the inherent noise in stochastic gradients.

Despite these previous assumptions, there exists empirical and theoretical evidence that using a large learning rate in the stochastic gradient bandit algorithm is a viable option. For example, it has been observed that softmax policies learn even with extremely large learning rates such as 2 14 [11]. For logistic regression on linearly separable data, the objective has an exponential tail and the minimizer is unbounded, yet it has been shown that gradient descent with iteration dependent learning rate η ∈ Θ( t ) achieves accelerated O (1 /t 2 ) convergence [35]. Though the objective in Eq. (2) has

similar properties, unlike logistic regression, the problem we are considering is non-concave, so the same techniques cannot be directly applied. The most related results are from [24], which proved that with a small problem specific constant learning rate, Algorithm 1 achieves convergence to a globally optimal policy almost surely. However, as mentioned, the learning rate choices in [24] rely on assumptions of (non-uniform) smoothness and noise growth conditions (their Lemmas 4.2, 4.3, and 4.6), which cannot be directly applied here for a large learning rate.

Consequently, the use of large learning rates appear to render existing results and techniques inapplicable. Furthermore, with a large constant learning rate, it is unclear whether Algorithm 1 will converge to any stationary point, or the iterates will keep oscillating. If the algorithm does converge, it is also not clear what effect large step-sizes have on exploration, and whether the algorithm will converge to the optimal arm in such cases. Resolving these questions requires new results that characterize the behavior of Algorithm 1, since the classical optimization and stochastic approximation convergence theories are no longer applicable, as explained.

3 Asymptotic Global Convergence of Gradient Bandit Algorithm

We have seen that solving the non-concave maximization problem Eq. (2) using Algorithm 1 with any constant (potentially large) learning rate requires ideas beyond classical optimization theory. Here, we take a different perspective to investigate how Algorithm 1 samples actions. For analysis, we make the following assumption about the reward distribution.

̸

Assumption 1 (True mean reward has no ties) . For all i, j ∈ [ K ] , if i = j , then r ( i ) = r ( j ) .

̸

Remark 1. Removing Assumption 1 remains an open question for future work, while we believe that Algorithm 1 works without Assumption 1. One piece of evidence to support this conjecture is that even in the exact gradient setting, the set of initializations where Softmax PG approaches non-strict one-hot policies has zero measure.

3.1 Failure Mode of Aggressive Updates

It has been observed that several accelerated PG methods in the true gradient setting, including natural PG [13, 1] and normalized PG [21], obtain worse results than standard softmax PG if combined with online sampling a t ∼ π θ t ( · ) using constant learning rates [20]. The failure mode in these cases is that the update is too aggressive and commits to a sub-optimal arm without sufficiently exploring all arms. This results in a non-trivial probability of sampling one action forever, i.e., there exists a potentially sub-optimal action a ∈ [ K ] , such that with some constant probability, a t = a for all t ≥ 1 . Such an outcome implies that π θ t ( a ) → 1 as t →∞ [20, Theorem 3]. Since a ∈ [ K ] could be a sub-optimal action with r ( a ) < r ( a ∗ ) = max a ∈ [ K ] r ( a ) , this results in a lack of exploration, and consequently, methods such as natural PG and normalized PG are not guaranteed to converge to the optimal action a ∗ := arg max a ∈ [ K ] r ( a ) with probability 1 .

3.2 Stochastic Gradient Automatically Avoids Lack of Exploration

Our first key finding is that Algorithm 1 does not keep sampling one action forever, no matter how large the constant learning rate is. This property avoids the problem of a lack of exploration, in the sense that Algorithm 1 will at least explore more than one action infinitely often. At first glance, this might not seem like a strong property, since the algorithm might somehow explore only sub-optimal actions forever. However, we will argue below that this property coupled with additional arguments is sufficient to guarantee convergence to the globally optimal policy.

Let us now formally prove the above property. By Algorithm 1, for all a ∈ [ K ] , for all t ≥ 1 ,

We define N t ( a ) as the number of times action a ∈ [ K ] is sampled up to iteration t ≥ 1 , i.e.,

and its asymptotic limit N ∞ ( a ) := lim t →∞ N t ( a ) , which could possibly be infinity. For all a ∈ [ K ] , we have either N ∞ ( a ) = ∞ or N ∞ ( a ) < ∞ , meaning that a ∈ [ K ] is sampled infinitely often or only finitely many times asymptotically. First, we prove the following Lemma 1, which shows that if an action a ∈ [ K ] is sampled only finitely many times as t →∞ , then the parameter corresponding to action a is also finite, i.e., sup t ≥ 1 | θ t ( a ) | < ∞ .

Lemma 1. Using Algorithm 1 with any constant η ∈ Θ(1) , if N ∞ ( a ) < ∞ for an action a ∈ [ K ] , then we have, almost surely,

Lemma 1 will be used multiple times in the subsequent convergence arguments.

̸

Proof sketch. Since we assume action a ∈ [ K ] is sampled finitely many times, the update given in the case depicted by Eq. (5) happens finitely many times. Each update is bounded since the sampled reward is in [ -R max , R max ] by Eq. (1), and the learning rate is a constant, i.e., η ∈ Θ(1) . In Algorithm 1, θ t ( a ) is still updated even when a t = a , with the corresponding update given by the case depicted by Eq. (6). Therefore, whether θ t ( a ) is bounded depends on the cumulative probability ∑ t s =1 π θ s ( a ) being summable as t → ∞ . According to the extended Borel-Cantelli lemma (Lemma 3), we have, almost surely,

which implies (by taking complements) that ∑ t ≥ 1 π θ t ( a ) < ∞ if and only if N ∞ ( a ) < ∞ . Therefore, if a ∈ [ K ] is sampled finitely often, θ t ( a ) will be updated in a bounded manner (using Eqs. (5) and (6)) as t →∞ , hence establishing Lemma 1. Detailed proofs for this lemma, as well as for all other results in this paper can be found in the appendix.

Given Lemma 1, we can then establish the above-mentioned finding about the exploration effect of Algorithm 1 in Lemma 2.

̸

Lemma 2 (Avoiding a lack of exploration) . Using Algorithm 1 with any η ∈ Θ(1) , there exists at least a pair of distinct actions i, j ∈ [ K ] and i = j , such that, almost surely,

Proof sketch. The argument for the existence of one such action is straightforward, since by the pigeonhole principle, if there are finitely many actions, i.e., K < ∞ , there must be at least one action i ∈ [ K ] that is sampled infinitely often as t →∞ .

̸

The argument for the existence of a second such action is by contradiction. Suppose that all the other actions j ∈ [ K ] with j = i are sampled only finitely many times as t → ∞ . According to Lemma 1, their corresponding parameters must remain finite, i.e., sup t ≥ 1 | θ t ( j ) | < ∞ for all j ∈ [ K ] with j = i . Now consider θ t ( i ) . By assumption, the second update case for this parameter, Eq. (6), happens only finitely often, since Eq. (6) can only occur when a t = i . Therefore, the key question is whether the cumulative probability ∑ t s =1 (1 -π θ s ( i )) involved in the first case of the update, Eq. (5), is summable as t →∞ . Note that ∑ t s =1 (1 -π θ s ( i )) = ∑ t s =1 ∑ j = i π θ s ( j ) , which is indeed summable as t →∞ , by the assumption and Eq. (9). This implies that action i , which is sampled infinitely often, achieves a parameter magnitude, sup t ≥ 1 | θ t ( i ) | < ∞ , that remains bounded as t →∞ . Using the softmax parameterization Eq. (3) in the above argument, we conclude that for all a ∈ [ K ] , inf t ≥ 1 π θ t ( a ) > 0 , i.e., every action's probability remains bounded away from zero, and hence is not summable. Using Eq. (9), this implies that every action is sampled infinitely often, which contradicts the assumption that only action i is sampled infinitely often as t →∞ .

̸

̸

Discussion. Lemma 2 implies that Algorithm 1 is not an aggressive method in the sense of [20], no matter how large the learning rate is, as long as it is constant, i.e., η ∈ Θ(1) . According to [20, Theorem 7], even if we fix the sampling in Algorithm 1 to a sub-optimal action a ∈ [ K ] forever, i.e., a t = a for all t ≥ 1 , its probability will not approach 1 faster than O (1 /t ) , i.e., 1 -π θ t ( a ) ∈ Ω(1 /t ) . This means that there must be at least one another action a ′ ∈ [ K ] with a ′ = a , such that a ′ will also be sampled infinitely often. A more intuitive explanation is that the (1 -π θ t ( a )) term in Eq. (5) will

̸

̸

be near 0 , which slows the speed of committing to a deterministic policy on a whenever π θ t ( a ) is close to 1 , which encourages exploration. Such natural exploratory behavior arises in Algorithm 1 because of the softmax Jacobian diag ( π θ ) -π θ π ⊤ θ in the update shown in Proposition 1, which determines the growth order of θ t ( a ) for all a ∈ [ K ] as t → ∞ , making the effect of a constant learning rate η ∈ Θ(1) asymptotically inconsequential.

3.3 Warm up: Global Asymptotic Convergence when K = 2

We now consider the simplest case, where we have only two possible actions. According to Lemma 2, each of the two actions must be sampled infinitely often as t →∞ . We now illustrate the second key result, that for both actions a ∈ [ K ] , the random sequence { θ t ( a ) } t ≥ 1 follows the direction of the expected gradient for sufficiently large t ≥ 1 almost surely. The proof uses a technique that has been previously used in [19, 24] for small learning rates, but here we observe that the same technique continues to work for Algorithm 1 no matter how large the learning rate is, as long as η ∈ Θ(1) .

Theorem 1. Let K = 2 and r (1) > r (2) . Using Algorithm 1 with any η ∈ Θ(1) , we have, almost surely, π θ t ( a ∗ ) → 1 as t →∞ , where a ∗ := arg max a ∈ [ K ] r ( a ) (equal to Action 1 in this case).

̸

Proof sketch. According to Lemma 2, N ∞ (1) = N ∞ (2) = ∞ . Denote the the reward gap as ∆ := r ( a ∗ ) -max a = a ∗ r ( a ) > 0 , which becomes ∆ = r (1) -r (2) for two actions. Since the stochastic gradient is unbiased (Proposition 1), we have, for all t ≥ 1 (detailed calculations omitted),

A similar calculation shows that,

which means that θ t ( a ∗ ) is monotonically increasing in expectation and θ t (2) is monotonically decreasing in expectation. In other words, { θ t ( a ∗ ) } t ≥ 1 is a sub-martingale, while { θ t (2) } t ≥ 1 is a super-martingale. However, since θ t ∈ R K is unbounded, Doob's martingale convergence results cannot be directly applied, so we pursue a different argument. Following [19, 24], given an action a ∈ [ K ] , we define P t ( a ) := E t [ θ t +1 ( a )] -θ t ( a ) as the 'progress', and define W t ( a ) := θ t ( a ) -E t -1 [ θ t ( a )] as the 'noise', where θ t ( a ) = W t ( a ) + P t -1 ( a ) + θ t -1 ( a ) . By recursion we can determine that,

i.e., θ t ( a ) is the result of 'cumulative progress' and 'cumulative noise'. According to [24, Theorem C.3], the cumulative noise term can be bounded by using martingale concentration, where the order of the corresponding confidence interval is smaller than the order of the cumulative progress. Therefore, the summation will always be determined by the cumulative progress as t → ∞ . According to the calculations in Eqs. (12) and (13), we have P t ( a ∗ ) > 0 and P t (2) < 0 , both of which are not summable. As a result, θ t ( a ∗ ) → ∞ and θ t (2) → -∞ as t → ∞ , which implies that π θ t ( a ∗ ) π θ t (2) = exp { θ t ( a ∗ ) -θ t ( a ) } → ∞ , hence π θ t ( a ∗ ) → 1 as t →∞ .

3.4 Global Asymptotic Convergence for all K ≥ 2

The illustrative two-action case shows that if π ⊤ θ t r ∈ ( r (2) , r ( a ∗ )) and if both actions are sampled infinitely often, then we have, almost surely θ t ( a ∗ ) →∞ and θ t (2) →-∞ as t →∞ . However, the question at the beginning of Section 3.2 remains: when K > 2 , if the two actions sampled infinitely often in Lemma 2 are both sub-optimal, will that result in a similar failure mode to the one described in Section 3.1? The answer is no, which follows from our third key finding, which is based on another contradiction-based argument that establishes almost sure convergence to a globally optimal policy in the general K > 2 case.

Theorem 2. Given K ≥ 2 , using Algorithm 1 with any η ∈ Θ(1) , we have, almost surely, π θ t ( a ∗ ) → 1 as t →∞ , where a ∗ = arg max a ∈ [ K ] r ( a ) is the optimal action.

Proof sketch. We consider two cases: N ∞ ( a ∗ ) < ∞ and N ∞ ( a ∗ ) = ∞ , corresponding to whether the optimal action is sampled finitely or infinitely often as t → ∞ . We argue that the first case ( N ∞ ( a ∗ ) < ∞ ) is impossible, while for the second case ( N ∞ ( a ∗ ) = ∞ ) we prove that θ t ( a ∗ ) -θ t ( a ) →∞ for all a ∈ [ K ] with r ( a ) < r ( a ∗ ) , which implies π θ t ( a ∗ ) → 1 as t →∞ .

̸

First case. Suppose that N ∞ ( a ∗ ) < ∞ . We argue that this is impossible via contradiction. Given the assumption and Lemma 2 we know there must be at least two other sub-optimal actions i 1 , i 2 ∈ [ K ] , i 1 = i 2 , such that N ∞ ( i 1 ) = N ∞ ( i 2 ) = ∞ . In particular, let i 1 = arg min a ∈ [ K ] ,N ∞ ( a )= ∞ r ( a ) and i 2 = arg max a ∈ [ K ] ,N ∞ ( a )= ∞ r ( a ) , hence r ( i 1 ) < r ( i 2 ) < r ( a ∗ ) . By Lemma 6 (see Appendix) we will also have r ( i 1 ) < π ⊤ θ t r < r ( i 2 ) for sufficiently large t ≥ 1 , which implies for action i 1 ,

for sufficiently large t ≥ 1 , which further implies that sup t ≥ 1 θ t ( i 1 ) < ∞ . Meanwhile, for the optimal action a ∗ , the assumption and Lemma 1 imply that inf t ≥ 1 θ t ( a ∗ ) > -∞ . Combining these two observations gives,

On the other hand, since N ∞ ( i 1 ) = ∞ and N ∞ ( a ∗ ) < ∞ (by assumption), we then have sup t ≥ 1 exp { θ t ( i 1 ) -θ t ( a ∗ ) } = ∞ by Lemma 5 (see Appendix), which contradicts Eq. (16).

̸

Second case. Suppose that N ∞ ( a ∗ ) = ∞ . We will argue that π θ t ( a ∗ ) → 1 as t →∞ almost surely. First, according to Lemma 2, there exists at least one sub-optimal action i 1 ∈ [ K ] , i 1 = a ∗ , such that N ∞ ( i 1 ) = ∞ . Let i 1 = arg min a ∈ [ K ] ,N ∞ ( a )= ∞ r ( a ) . By Lemma 6 and the definition of a ∗ , we have r ( i 1 ) < π ⊤ θ t r < r ( a ∗ ) for all sufficiently large t ≥ 1 . Since N ∞ ( i 1 ) = ∞ , using similar calculations to Eqs. (13) and (14) in Theorem 1, we have, θ t ( i 1 ) →-∞ as t →∞ . We also have, inf t ≥ 1 θ t ( a ∗ ) > -∞ as t →∞ . Hence, π θ t ( a ∗ ) π θ t ( i 1 ) = exp { θ t ( a ∗ ) -θ t ( i 1 ) } → ∞ as t →∞ .

Define A ∞ := { a ∈ [ K ] | N ∞ ( a ) = ∞} as the set of actions that are sampled infinitely often, and note that |A ∞ | ≥ 2 by Lemma 2. Sort the action indices in A ∞ according to their expected reward values in descending order, i.e.,

Assumption 1 is used here to prevent two arms from having the same reward and thus guarantee the inequalities are strict in Eq. (17). Next, using similar calculations as in Lemma 6, we have,

̸

where A -( i 2 ) := { a -∈ [ K ] : r ( a -) < r ( i 2 ) } is the set of actions that have lower mean reward than i 2 ∈ [ K ] , and note that i 1 ∈ A -( i 2 ) . Using the above definitions, we can conclude that i 1 is the only arm in A -( i 2 ) that has been sampled infinitely often. According to Lemma 5, for all a -∈ A -( i 2 ) with a -= i 1 , we have, π θ t ( a ∗ ) π θ t ( a -) → ∞ as t → ∞ , since N ∞ ( a ∗ ) = ∞ (by assumption) and N ∞ ( a -) < ∞ (by Eq. (17)). Therefore, for all sufficiently large t , the probability ratio in Eq. (18) π θ t ( a ∗ ) π θ t ( a -) →∞ for all a -∈ A -( i 2 ) , which implies that, for all sufficiently large t ≥ 1 ,

We have thus shown that π ⊤ θ t r > r ( i 2 ) . Recall that we had previously proved that π ⊤ θ t r > r ( i 1 ) . Hence, we will apply this argument recursively: after this point, i 2 ∈ [ K ] will become the new ' i 1 ∈ [ K ] ', and a similar inequality to Eq. (13) will then hold for i 2 ∈ [ K ] from similar calculations to Eqs. (13) and (14) in Theorem 1, establishing θ t ( i 2 ) →-∞ as t →∞ . This will imply that for all sufficiently large t ≥ 1 ,

̸

Continuing the recursive argument, we can conclude for all actions a ∈ A ∞ with a = a ∗ that π θ t ( a ∗ ) π θ t ( a ) →∞ as t →∞ . Meanwhile, for all actions a ̸∈ A ∞ , Lemma 5 also shows that π θ t ( a ∗ ) π θ t ( a ) →∞ as t →∞ . Combining these two results yields the conclusion that for all sub-optimal actions a ∈ [ K ] with r ( a ) < r ( a ∗ ) we have π θ t ( a ∗ ) π θ t ( a ) →∞ as t →∞ , which implies π θ t ( a ∗ ) → 1 as t →∞ . Thus, we have established almost sure convergence to the globally optimal policy.

Discussion. Lemma 2 is important to prove that the optimal arm will be sampled infinitely often. In particular, Lemma 2 guarantees |A ∞ | ≥ 2 and the existence of i 1 in Eq. (15), which can then be used to construct the contradiction in Eq. (16). Without Lemma 2, |A ∞ | might be equal to 1 and the failure mode in Section 3.1 can occur, resulting in Algorithm 1 not sampling the optimal action infinitely often as t →∞ .

3.5 Asymptotic Rate of Convergence

According to Theorem 2, almost surely, π θ t ( a ∗ ) → 1 as t →∞ . Therefore, after a large enough time τ < ∞ , we have π θ t ( a ∗ ) ≥ 1 / 2 , which implies that the 'progress' term in Eq. (14) can be lower bounded. With this, an asymptotic rate of convergence can be proved as follows.

Theorem 3. For a large enough τ > 0 , for all T > τ , the average sub-optimality decreases at an O ( ln( T ) T ) rate. Formally, if a ∗ is the optimal arm, then, for a constant c ,

4 Simulation Study

To validate and enhance the theoretical findings, we ran experiments with a four action bandit environment ( K = 4 ) with a true mean reward vector of r = (0 . 2 , 0 . 05 , -0 . 1 , -0 . 4) ⊤ ∈ R 4 . The reward distribution P a for arm 1 ≤ a ≤ 4 is Gaussian, centered at r ( a ) and with a standard deviation of 0 . 1 . The environment is chosen to illustrate various phenomenon, which we discuss after presenting the results. The algorithm is Algorithm 1 with θ 1 = 0 ∈ R K .

For comparison, assuming that the random rewards belong to the [ -1 , 1] interval, the only result for the stochastic gradient bandit algorithm [24, Lemma 4.6] that allowed a constant learning rate required that the learning rate be less than η c = ∆ 2 40 · K 3 / 2 · R 3 max = 9 128000 ≈ 0 . 00007 , where we used ∆ = 0 . 15 , K = 4 , and R max = 1 . While technically, the result does not apply to our case where the reward distributions have unbounded support, the probability of the reward landing outside of [ -1 , 1] is in the order of 10 -9 . Choosing R max to be larger, this probability falls extremely quickly, which suggests that the above threshold is generous. For the experiments we use the learning rates η ∈ { 1 , 10 , 100 , 1000 } , that are several orders of magnitudes larger than η c . For each learning rate, we plot the outcome of 10 runs, corresponding to different random seeds. Each run lasts 10 6 ( ≈ e 14 ) iterations. The log-suboptimality gaps for the 4 × 10 cases are shown on Figures 1a-1d, where they are plotted against the logarithm of time. Additional results for K = 2 arms are shown in Appendix D. Note that in this example small sub-optimality implies that the optimal arm is chosen with high probability. In what follows, we discuss the results in the plots.

Asymptotic convergence. For the smaller learning rates of η = 1 and 10 , all 10 seeds rapidly and steadily converge, reaching a sub-optimality of e -14 or less. For η = 100 and 1000 , most of the runs reach even small error even faster, but some runs are 'stuck' even after 10 6 steps. Note that this does not contradict the theoretical result; nor do we suspect numerical issues. As seen for the case of η = 100 , even after a long phase with little to no progress, a run can 'recover' (see the grey curve). In fact, it is reasonable to expect that the price of increasing the learning rate is larger variance; as seen in these plots (subplots (a) and (b) are also attesting to this). Differences between learning rates are further discussed below.

Non-monotone objective value. Using a very small learning rate guarantees monotonic improvement (in expectation) in the policy's expected reward [24]. Conversely, a large learning rate results in non-monotonic evolution of the expected rewards { π ⊤ θ t r } t ≥ 1 , even in the final stages of convergence, as can be seen clearly for the learning rates of η = 1 and 10 in Figures 1a and 1b. For larger η , the non-monotone behavior happens over longer periods and is less visible in the plots. This is because using large learning rates causes the policy to rapidly increase the parameters for some action, after which the gradient becomes small, limiting further progress.

Rate of convergence. Figures 1a and 1b, where the log-log plot has a slope of nearly -1 , give some evidence that an O (1 /t ) asymptotic rate is achieved. In general, such a rate cannot be improved

0

2

4

6

8

10

12

14

log(t)

14

12

10

8

6

4

2

Log sub-optimality gap

(a)

η

= 1

.

0

2

4

6

8

10

12

14

log(t)

20.0

17.5

15.0

12.5

10.0

7.5

5.0

2.5

Log sub-optimality gap

(b)

η

= 10

.

0

2

4

6

8

10

12

14

log(t)

30

25

20

15

10

5

0

Log sub-optimality gap

(c)

η

= 100

.

0

2

4

6

8

10

12

14

log(t)

300

250

200

150

100

50

0

Log sub-optimality gap

(d)

η

= 1000

.

in terms of t [14]. Theorem 3 gives a weaker version of convergence rate over averaged iterates (not last iterate), which is slightly worse than O (1 /t ) . More work is needed to verify if the asymptotic convergence rate in Theorem 3 is improvable or not.

Different learning rates. Two observations can be made from Figure 1 regarding the effect of using different η values: First , during the final stage of convergence when r ( a ∗ ) -π ⊤ θ t r ≈ 0 , using larger η results in faster convergence on average. As η increases, the order of log ( r ( a ∗ ) -π ⊤ θ t r ) also changes from e -14 ( η = 1 ), to e -20 ( η = 100 ), and e -200 ( η = 1000 ). We conjecture that the asymptotic rate of convergence has an O (1 /η ) dependence. Second , using larger learning rates can take a longer time to enter the final stage of convergence. When η = 1 or 10 , all curves quickly enter the final stage of r ( a ∗ ) -π ⊤ θ t r ≈ 0 . However, for larger η values, 1 / 10 runs ( η = 100 ) and 3 / 10 runs ( η = 1000 ) result in r ( a ∗ ) -π ⊤ θ t r values far from 0 even after 10 6 iterations. These runs take orders of magnitude more iterations to eventually achieve r ( a ∗ ) -π ⊤ θ t r ≈ 0 . These situations correspond to the policy π θ t getting stuck near sub-optimal corners of the simplex, meaning that π θ t ( i ) ≈ 1 for a sub-optimal action i ∈ [ K ] with r ( i ) < r ( a ∗ ) . In such cases, even Softmax PG with the true gradient can remain stuck on a sub-optimal plateau for an extremely long time [23]. However, the reason why larger learning rates lead to longer plateaus in the stochastic setting remains unclear.

Trade-offs and multi-stage chracterizations of convergence. Given the above observations, there appears to exist a trade-off for η : larger η values result in faster convergence during the final stage where r ( a ∗ ) -π ⊤ θ t r ≈ 0 , but at the the cost of taking far longer to enter this final stage of convergence. Since asymptotic convergence results are insufficient for explaining these subtleties in a satisfactory manner, a more refined analysis that considers the different stages of convergence is required.

5 Conclusions and Future Directions

This work refines our understanding of stochastic gradient bandit algorithms by proving that it converges to a globally optimal policy almost surely with any constant learning rate. Our new proof strategy based on the asymptotics of sample counts opens new directions for better characterizing exploration effects of stochastic gradient methods, while also suggesting interesting new questions. Characterizing the multiple stages of convergence remains another interesting future direction. One interesting possibility is that there might exist an optimal time-dependent scheme for increasing the learning rate (such as η ∈ O (log t ) ) to accelerate convergence, rather than use a constant η ∈ O (1) . This is corroborated by our experiments: As seen in Figure 1, small learning rates perform better during the early stages of optimization, while larger learning rates achieve faster convergence during the final stage. Other directions include extending our bandit results to the more general RL setting [34], as well as extending our results for the softmax tabular parameterization to handle function approximation [1].

Limitations: While this work establishes a surprising asymptotic convergence result for any constant learning rate, it does not shed light on the effect of different learning rates on the convergence. Moreover, our analysis is limited to multi-armed bandits, and does not immediately extend to the general RL setting. These aspects are the main limitations of this paper.

Broader impact: This is primarily theoretical work on a fundamental algorithm that is used broadly in RL applications. We expect these results to improve the research community's understanding of the basic stochastic gradient bandit method.

Acknowledgments and Disclosure of Funding

Jincheng Mei would like to thank Ramki Gummadi for providing feedback on a draft of this manuscript. Csaba Szepesvári and Dale Schuurmans gratefully acknowledge funding from the Canada CIFAR AI Chairs Program, Amii and NSERC. Sharan Vaswani acknowledges the support from the NSERC Discovery Grant (RGPIN-2022-04816).

References

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research , 22(98):1-76, 2021.

Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory , pages 39-1. JMLR Workshop and Conference Proceedings, 2012.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2):235-256, 2002.

Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers , pages 177-186. Springer, 2010.

Leo Breiman. Probability . SIAM, 1992.

Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research , 70(4):2563-2578, 2022.

Nicolò Cesa-bianchi and Claudio Gentile. Improved risk tail bounds for on-line algorithms. In Y. Weiss, B. Schölkopf, and J. Platt, editors, Advances in Neural Information Processing Systems , volume 18. MIT Press, 2005.

Denis Denisov and Neil Walton. Regret analysis of a markov policy gradient algorithm for multi-arm bandits. arXiv preprint arXiv:2007.10229 , 2020.

Yuhao Ding, Junzi Zhang, and Javad Lavaei. Beyond exact gradients: Convergence of stochastic soft-max policy gradient methods with entropy regularization. arXiv preprint arXiv:2110.10117 , 2021.

David A. Freedman. On Tail Probabilities for Martingales. The Annals of Probability , 3(1):100 - 118, 1975.

Shivam Garg, Samuele Tosatto, Yangchen Pan, Martha White, and A Rupam Mahmood. An alternate policy gradient estimator for softmax policies. arXiv preprint arXiv:2112.11622 , 2021.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

Sham M Kakade. A natural policy gradient. In Advances in neural information processing systems , pages 1531-1538, 2002.

Tze Leung Lai, Herbert Robbins, et al. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical programming , 198(1):1059-1106, 2023.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Softmax policy gradient methods can take exponential time to converge. In Conference on Learning Theory , pages 3107-3110. PMLR, 2021.

Michael Lu, Matin Aghaei, Anant Raj, and Sharan Vaswani. Towards principled, practical policy gradient for bandits and tabular mdps. arXiv preprint arXiv:2405.13136 , 2024.

Jincheng Mei, Wesley Chung, Valentin Thomas, Bo Dai, Csaba Szepesvari, and Dale Schuurmans. The role of baselines in policy gradient optimization. Advances in Neural Information Processing Systems , 35:17818-17830, 2022.

Jincheng Mei, Bo Dai, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. Understanding the effect of stochasticity in policy optimization. Advances in Neural Information Processing Systems , 34:19339-19351, 2021.

Jincheng Mei, Yue Gao, Bo Dai, Csaba Szepesvari, and Dale Schuurmans. Leveraging non-uniformity in first-order non-convex optimization. In International Conference on Machine Learning , pages 7555-7564. PMLR, 2021.

Jincheng Mei, Chenjun Xiao, Bo Dai, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Escaping the gravitational pull of softmax. Advances in Neural Information Processing Systems , 33:21130-21140, 2020.

Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference on Machine Learning , pages 6820-6829. PMLR, 2020.

Jincheng Mei, Zixin Zhong, Bo Dai, Alekh Agarwal, Csaba Szepesvari, and Dale Schuurmans. Stochastic gradient succeeds for bandits. arXiv preprint arXiv:2402.17235 , 2024.

Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization , 19(4):1574-1609, 2009.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024.

Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics , pages 400-407, 1951.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897, 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction . MIT Press, 2018.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999.

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8:229-256, 1992.

Jingfeng Wu, Peter L Bartlett, Matus Telgarsky, and Bin Yu. Large stepsize gradient descent for logistic loss: Non-monotonicity of the loss improves optimization efficiency. arXiv preprint arXiv:2402.15926 , 2024.

Rui Yuan, Robert M Gower, and Alessandro Lazaric. A general sample complexity analysis of vanilla policy gradient. In International Conference on Artificial Intelligence and Statistics , pages 3332-3380. PMLR, 2022.

Junyu Zhang, Chengzhuo Ni, Csaba Szepesvari, Mengdi Wang, et al. On the convergence and sample efficiency of variance-reduced policy gradient method. Advances in Neural Information Processing Systems , 34:2228-2240, 2021.

Junzi Zhang, Jongho Kim, Brendan O'Donoghue, and Stephen Boyd. Sample efficient reinforcement learning with reinforce. arXiv preprint arXiv:2010.11364 , 2020.

Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization , 58(6):3586-3612, 2020.

A Asymptotic Convergence

Lemma 1. Using Algorithm 1 with any constant η ∈ Θ(1) , if N ∞ ( a ) < ∞ for an action a ∈ [ K ] , then we have, almost surely,

Proof. Suppose N ∞ ( a ) < ∞ for an action a ∈ [ K ] . According to Algorithm 1,

and let

According to Eq. (22), we have, for all t ≥ 1 ,

Using triangle inequality, we have,

By assumption, we have, which implies that, almost surely,

̸

Lemma 2 (Avoiding lack of exploration) . Using Algorithm 1 with any η ∈ Θ(1) , there exists at least a pair of distinct actions i, j ∈ [ K ] and i = j , such that, almost surely,

According to the extended Borel-Cantelli Lemma 3, we have, almost surely,

Combining Eqs. (25), (28) and (29), we have, almost surely,

Proof. First, we have, for all t ≥ 1 ,

By pigeonhole principle, there exists at least one action i ∈ [ K ] , such that, almost surely,

We argue the existence of another action by contradiction. Suppose for all the other actions j ∈ [ K ] and j = i , we have N ∞ ( j ) < ∞ . According to the extended Borel-Cantelli Lemma 3, we have, almost surely,

According to the update Eq. (22), we have, for all t ≥ 1 ,

By triangle inequality, we have,

̸

̸

̸

̸

̸

Combing Eqs. (36) and (38) and the assumption of N ∞ ( j ) < ∞ for all j = i , we have, almost surely,

̸

Since N ∞ ( j ) < ∞ for all j = i by assumption, and according to Lemma 1, we have, almost surely,

Combining Eqs. (42) and (43), we have, for all action a ∈ [ K ] ,

which implies that, there exists c > 0 and c ∈ O (1) , such that, for all a ∈ [ K ] ,

̸

Therefore, for all action a ∈ [ K ] ,

̸

According to the extended Borel-Cantelli Lemma 3, we have, almost surely, for all action a ∈ [ K ] ,

̸

which is a contradiction with the assumption of N ∞ ( j ) < ∞ for all the other actions j ∈ [ K ] with j = i . Therefore, there exists another action j ∈ [ K ] with j = i , such that N ∞ ( j ) = ∞ .

Theorem 1. Let K = 2 and r (1) > r (2) . Using Algorithm 1 with any η ∈ Θ(1) , we have, almost surely, π θ t ( a ∗ ) → 1 as t →∞ , where a ∗ := arg max a ∈ [ K ] r ( a ) (equal to Action 1 in this case).

Proof. The proof uses the same notations as of [24, Theorem 5.1].

According to Lemma 2, we have, N ∞ (1) = ∞ and N ∞ (2) = ∞ , i.e., both of the two actions are sampled for infinitely many times as t →∞ .

Let F t be the σ -algebra generated by a 1 , R 1 ( a 1 ) , · · · , a t -1 , R t -1 ( a t -1 ) :

Note that θ t and I t (defined by Eq. (23)) are F t -measurable for all t ≥ 1 . Let E t [ · ] denote the conditional expectation with respect to F t : E t [ X ] = E [ X |F t ] . Define the following notations,

For each action a ∈ [ K ] , for t ≥ 2 , we have the following decomposition,

By recursion we can determine that,

and E [ θ 1 ( a )] accounts for potential randomness in initializing θ 1 ∈ R K . According to Proposition 1, we have,

which implies that, for all t ≥ 1 , while we also have,

Next, we show that ∑ t s =1 P s ( a ∗ ) →∞ as t →∞ by contradiction. Suppose that,

For the optimal action a ∗ = 1 ,

̸

where ∆ := r ( a ∗ ) -max a = a ∗ r ( a ) = r (1) -r (2) > 0 is the reward gap. Denote that, for all t ≥ 1 ,

According to Lemma 8 (using Eqs. (58) to (60) and (62)), we have, almost surely,

For the sub-optimal action (Action 2 ), we have,

and also denote, for all t ≥ 1 ,

According to Lemma 8 (using Eqs. (58), (59), (64) and (67)), we have, almost surely,

Combining Eqs. (63) and (68), there exists c > 0 and c ∈ O (1) , such that, for all a ∈ { a ∗ , 2 } ,

which implies that,

which is a contradiction with the assumption of Eq. (59). Therefore, we have,

According to Lemma 9 (using Eqs. (58), (60), (62) and (74)), we have, almost surely,

Similarly, according to Lemma 10 (using Eqs. (58), (64), (67) and (74)), we have, almost surely,

Combining Eqs. (75) to (77), we have, almost surely,

Theorem 2. Under Assumption 1, given K ≥ 2 , using Algorithm 1 with any η ∈ Θ(1) , we have, almost surely, π θ t ( a ∗ ) → 1 as t →∞ , where a ∗ = arg max a ∈ [ K ] r ( a ) is the optimal action.

Proof. Similar to the proof of Theorem 1, we define F t to be the σ -algebra generated by a 1 , R 1 ( a 1 ) , · · · , a t -1 , R t -1 ( a t -1 ) i.e.

Note that θ t and I t (defined by Eq. (23)) are F t -measurable for all t ≥ 1 . Let E t [ · ] denote the conditional expectation with respect to F t : E t [ X ] = E [ X |F t ] . Define the following notations,

For each action a ∈ [ K ] , for t ≥ 2 , we have the following decomposition,

By recursion we can determine that,

and E [ θ 1 ( a )] accounts for potential randomness in initializing θ 1 ∈ R K . For an action a ∈ [ K ] ,

Define A ∞ as the set of actions which are sampled for infinitely many times as t →∞ , i.e.,

First part. We show that N ∞ ( a ∗ ) = ∞ by contradiction.

Suppose a ∗ ̸∈ A ∞ i.e. N ∞ ( a ∗ ) < ∞ . According to Lemma 2, we have, |A ∞ | ≥ 2 . Since a ∗ ̸∈ A ∞ by assumption, there must be at least two other sub-optimal actions i 1 , i 2 ∈ [ K ] , i 1 = i 2 , such that,

̸

Note that, while we also have,

In particular, define

Using the update for arm i 1 , we know that, for all s ≥ 1 ,

By definition and because of Assumption 1, we have that r ( i 1 ) < r ( i 2 ) < r ( a ∗ ) . Furthermore, according to Lemma 6, we have, for sufficiently large τ ≥ 1 ,

which implies that after some large enough τ ≥ 1 ,

̸

From Eq. (89), we have,

which implies that, for all a -∈ A -( i 1 ) , we have,

Note that, by definition, i 2 ∈ A + ( i 1 ) , and

In order to bound Eq. (95) using the above relations, note that,

(Fewer terms in the denominator)

By Lemma 5, for large enough τ ≥ 1 , for all a -, π θs ( a -) π θs ( i 2 ) ≤ 1 |A -( i 1 ) | · r ( i 2 ) -r ( i 1 ) r ( i 1 ) -r ( a -) . Hence,

Combining Eqs. (94), (95) and (102), we have,

where

̸

Bounding the variance for arm i 1 , we have that for all large enough τ ≥ 1 ,

Using similar calculations as for the progress term, we have,

By Lemma 5, for large enough τ ≥ 1 , for all a -, π θs ( a -) π θs ( i 2 ) ≤ 13 5 1 |A -( i 1 ) | . Hence,

Combining Eqs. (108) and (111), we have that for large enough τ ≥ 1 ,

According to Lemma 12 (using Eqs. (94), (105) and (113)), we have, almost surely,

Since N ∞ ( a ∗ ) < ∞ by assumption, and according to Lemma 1, we have, almost surely,

Combining Eqs. (114) and (115), we have,

On the other hand, by Lemma 5, we have,

which contradicts Eq. (116). Hence, the assumption that N ∞ ( a ∗ ) < ∞ cannot hold. This completes the proof by contradiction, and implies that N ∞ ( a ∗ ) = ∞ .

Second part. With a ∗ ∈ A ∞ i.e. N ∞ ( a ∗ ) = ∞ , we now argue that π θ t ( a ∗ ) → 1 as t →∞ almost surely.

According to Lemma 2, we have, |A ∞ | ≥ 2 . Since a ∗ ∈ A ∞ by assumption, there must be at least one sub-optimal action i 1 ∈ [ K ] with r ( i 1 ) < r ( a ∗ ) , such that,

According to Lemma 6, we have, for all sufficiently large τ ≥ 1 ,

Using the same arguments as in the first part (except that i 2 in Eq. (98) is replaced with a ∗ ), both Eqs. (105) and (113) hold.

Next, we argue that ∑ t s = τ π θ s ( i 1 ) ∑ a + ∈A + ( i 1 ) π θ s ( a + ) →∞ as t →∞ by contradiction.

Suppose

In particular, define

According to Lemma 8 (using Eqs. (94), (105), (113) and (121)), we have,

Calculating the progress and variance for arm a ∗ , for t ≥ 1 ,

̸

Denote that, for all t ≥ 1 ,

According to Lemma 11 (using Eqs. (123) and (126)), we have,

Combining Eqs. (122) and (127), we have,

For all t ≥ 1 , | θ t ( i 1 ) | < ∞ and since there is at least one arm ( a ∗ ) s.t. inf t ≥ 1 θ t ( a ) > -∞ , there exists ϵ > 0 and ϵ ∈ O (1) , such that,

According to Eq. (118), we know that N ∞ ( i 1 ) = ∞ and for all a -∈ A -( i 1 ) , N ∞ ( a -) < ∞ . Using Lemma 5, we have, for all large enough t ≥ 1 , π θ t ( a -) π θ t ( i 1 ) < ϵ |A -( i 1 ) | .

Combining Eqs. (129) and (131), we have, for all large enough t ≥ 1 ,

which implies that,

which contradicts the assumption of Eq. (121). This completes the proof by contradiction, and therefore, we have,

According to Lemma 10 (using Eqs. (105), (113) and (135)), we have, almost surely,

Combining Eqs. (127) and (136), we have, almost surely,

Hence, we have proved that if N ∞ ( i 1 ) = ∞ and r ( i 1 ) < π ⊤ θ t r < r ( a ∗ ) for sufficiently large τ ≥ 1 , then, π θ t ( a ∗ ) π θ t ( i 1 ) →∞ , as t →∞ .

In order to use this argument recursively, consider sorting the action indices in A ∞ according to their descending expected reward values,

We know that,

̸

̸

According to Lemma 5, for all a -∈ A -( i 2 ) with a -= i 1 , we have,

Combining Eqs. (137) and (143), we have, for all a -∈ A -( i 2 ) ,

Hence, for all sufficiently large τ ≥ 1 , for all a -∈ A -( i 2 ) ,

Combining Eqs. (142) and (145), we have, for all sufficiently large t ≥ 1 ,

Hence we have, for all sufficiently large τ ≥ 1 ,

Comparing Eqs. (120) and (147), we can use a similar argument for i 2 and conclude that, for sufficiently large τ ≥ 1 , then, π θ t ( a ∗ ) π θ t ( i 2 ) →∞ , as t →∞ . This further implies that

̸

Continuing this recursive argument, we have, for all actions a ∈ A ∞ with a = a ∗ ,

Meanwhile, according to Lemma 5, we have, for all actions a ̸∈ A ∞ ,

Combining Eqs. (149) and (150), we have, for all sub-optimal actions a ∈ [ K ] with r ( a ) < r ( a ∗ ) ,

Combining Eqs. (151) and (153), we have, almost surely,

Finally, note that,

B Miscellaneous Extra Supporting Results

Lemma 3 (Extended Borel-Cantelli Lemma, Corollary 5.29 of [5]) . Let ( F n ) n ≥ 1 be a filtration, A n ∈ F n . Then, almost surely,

Lemma 4 (Freedman's inequality [10, 7], Theorem C.3 of [24]) . Let X 1 , X 2 , . . . be a sequence of random variables, such that for all t ≥ 1 , | X t | ≤ 1 / 2 . Define

Then, for all δ > 0 ,

̸

Lemma 5. Using Algorithm 1, for any two different actions i, j ∈ [ K ] with i = j , if N ∞ ( i ) = ∞ and N ∞ ( j ) < ∞ , then we have, almost surely,

Proof. We prove the result by contradiction. Suppose

According to the extended Borel-Cantelli Lemma 3, we have, for all a ∈ [ K ] , almost surely,

Hence, taking complements, we have,

also holds almost surely, which implies that,

which is a contradiction with Eq. (161).

Therefore, we have,

Lemma 6. Using Algorithm 1, we have, almost surely, for all large enough t ≥ 1 ,

where i 1 , i 2 ∈ [ K ] and i 1 = i 2 are the action indices defined as,

Proof. Define A ∞ as the set of actions which are sampled for infinitely many times as t →∞ , i.e.,

̸

According to Lemma 2, we have |A ∞ | ≥ 2 , which implies that i 1 = i 2 .

Given any sub-optimal action i ∈ [ K ] with r ( i ) < r ( a ∗ ) , we partition the remaining actions into two parts using r ( i ) .

By definition, we have,

First part. r ( i 1 ) < π ⊤ θ t r .

̸

If r ( i 1 ) = min a ∈ [ K ] r ( a ) , i.e., i 1 is the 'worst action', then r ( i 1 ) < π ⊤ θ t r holds trivially. Otherwise, suppose r ( i 1 ) = min a ∈ [ K ] r ( a ) . We have,

̸

Consider the non-empty set A ∞ ∩ A + ( i 1 ) . Pick an action j 1 ∈ A ∞ ∩ A + ( i 1 ) , and ignore all the other actions a + ∈ A + ( i 1 ) with a + = j 1 in the above equation. We have,

where the first inequality is because of r ( a + ) -r ( i 1 ) > 0 for all a + ∈ A + ( i 1 ) by Eq. (170). Note that N ∞ ( j 1 ) = ∞ and N ∞ ( a -) < ∞ . According to Lemma 5, we have, for all large enough t ≥ 1 ,

Combining Eqs. (175) and (177), we have,

̸

where the last inequality is because j 1 ∈ A + ( i 1 ) .

Second part. π ⊤ θ t r < r ( i 2 ) .

The arguments are similar to the first part. If r ( i 2 ) = r ( a ∗ ) , i.e., i 2 is the optimal action, then π ⊤ θ t r < r ( i 2 ) holds trivially. Otherwise, suppose r ( i 2 ) = r ( a ∗ ) . We have,

̸

̸

Consider the non-empty set A ∞ ∩ A -( i 2 ) . Pick an action j 2 ∈ A ∞ ∩ A -( i 2 ) , and ignore all the other actions a -∈ A -( i 2 ) with a -= j 2 in the above equation. We have,

where the first inequality is because of r ( i 2 ) -r ( a -) > 0 for all a -∈ A -( i 2 ) by Eq. (170). Note that N ∞ ( j 2 ) = ∞ and N ∞ ( a + ) < ∞ . According to Lemma 5, we have, for all large enough t ≥ 1 ,

Combining Eqs. (182) and (184), we have,

-

where the last inequality is because j 2 ∈ A ( i 2 ) .

For the following lemmas, we will use the notation defined in the proofs for Theorems 1 and 2. Lemma 7 (Concentration of noise) . Given an action a ∈ [ K ] . We have, with probability at least 1 -δ ,

where and

is originally defined by Eq. (52) .

Proof. First, note that,

Using the update Eq. (22), we have,

According to Eq. (1), we have,

The conditional variance of noise is,

where the inequality is by ( a + b ) 2 ≤ 2 a 2 +2 b 2 . Next, we have,

̸

̸

̸

̸

Combining Eqs. (196), (198) and (201), we have,

Let X t +1 ( a ) := W t +1 ( a ) 6 η · R max . Then we have,

According to Lemma 4, there exists an event E 1 such that Pr( E 1 ) ≥ 1 -δ , and when E 1 holds,

which implies that,

where V t ( a ) := 5 18 · ∑ t s =1 π θ s ( a ) · (1 -π θ s ( a )) .

and,

Lemma 8 (Bounded progress) . For any action a ∈ [ K ] with N ∞ ( a ) = ∞ , if there exists c > 0 and τ < ∞ , such that, for all t ≥ τ ,

where V t ( a ) is defined in Eq. (189) , and if also

Proof. According to Eq. (55) and triangle inequality, we have,

According to Lemma 7, there exists an event E 1 , such that Pr( E 1 ) ≥ 1 -δ , and when E 1 holds, we have, for all t ≥ τ +1 ,

According to Eq. (212), as t →∞ ,

According to Eqs. (211), (215) and (216), we have,

Combining Eqs. (214), (216) and (217), we have, as t →∞ ,

Take any ω ∈ E := { N ∞ ( a ) = ∞} . Because P ( E \ ( E ∩ E 1 )) ≤ P (Ω \ E 1 ) ≤ δ → 0 as δ → 0 , we have that P -almost surely for all ω ∈ E there exists δ > 0 such that ω ∈ E ∩ E 1 while Eq. (215) also holds for this δ . Take such a δ . We have, almost surely,

Lemma 9 (Unbounded positive progress) . For any action a ∈ [ K ] with N ∞ ( a ) = ∞ , if there exists c > 0 and τ < ∞ , such that, for all t ≥ τ ,

where V t ( a ) is defined in Eq. (189) , and if also,

then we have, almost surely, then we have, almost surely,

Proof. According to Eq. (55),

According to Lemma 7, there exists an event E 1 , such that Pr( E 1 ) ≥ 1 -δ , and when E 1 holds, we have, for all t ≥ τ +1 ,

By Eq. (221), as t →∞ , then we have, almost surely,

Proof.

and the speed of ∑ t -1 s =1 P s ( a ) → ∞ is strictly faster than ♡ → ∞ , according to Eq. (220). This implies that θ t ( a ) →∞ , as a result of 'cumulative progress' dominates 'cumulative noise'.

Take any ω ∈ E := { N ∞ ( a ) = ∞} . Because P ( E \ ( E ∩ E 1 )) ≤ P (Ω \ E 1 ) ≤ δ → 0 as δ → 0 , we have that P -almost surely for all ω ∈ E there exists δ > 0 such that ω ∈ E ∩ E 1 while Eq. (224) also holds for this δ . Take such a δ . We have, almost surely,

Lemma 10 (Unbounded negative progress) . For any action a ∈ [ K ] with N ∞ ( a ) = ∞ , if there exists c > 0 and τ < ∞ , such that, for all t ≥ τ ,

where V t ( a ) is defined in Eq. (189) , and if also,

The proof follows almost the same arguments for Lemma 9.

Lemma 11 (Positive progress) . For any action a ∈ [ K ] with N ∞ ( a ) = ∞ , if there exists c > 0 and τ < ∞ , such that, for all t ≥ τ ,

where V t ( a ) is defined in Eq. (189) , then we have, almost surely,

Proof. First case: if ∑ ∞ t =1 P t ( a ) < ∞ , then according to Lemma 8, we have, almost surely,

which implies Eq. (232).

Second case: if ∑ ∞ t =1 P t ( a ) = ∞ , then according to Lemma 9, we have, almost surely,

which also implies Eq. (232).

Lemma 12 (Negative progress) . For any action a ∈ [ K ] with N ∞ ( a ) = ∞ , if there exists c > 0 and τ < ∞ , such that, for all t ≥ τ ,

where V t ( a ) is defined in Eq. (189) , then we have, almost surely,

Proof. First case: if -∑ ∞ t =1 P t ( a ) < ∞ , then according to Lemma 8, we have, almost surely,

which implies Eq. (237).

Second case: if -∑ ∞ t =1 P t ( a ) = ∞ , then according to Lemma 10, we have, almost surely,

which also implies Eq. (237).

C Rate of Convergence

Theorem 3. For a large enough τ > 0 , for all T > τ , the average sub-optimality decreases at an O ( ln( T ) T ) rate. Formally, if a ∗ is the optimal arm, then, for a constant c

Proof. The progress for the optimal action is,

̸

Since lim t →∞ π θ t ( a ∗ ) = 1 , we have for all large enough t ≥ 1 ,

Since N ∞ ( a ∗ ) = ∞ (Theorem 2 ) and |A ∞ | ≥ 2 (Lemma 2), we have,

where N ∞ ( i 1 ) = ∞ and r ( i 1 ) < r ( a ∗ ) . Therefore, we have,

which will be dominated by sum of P t ( a ∗ ) since V t ( a ∗ ) appears under square root (Lemma 7). Therefore, for all large enough t ≥ τ ,

On the other hand, we argued recursively (in the proofs for Theorem 2) that for all sub-optimal action a ∈ [ K ] with r ( a ) < r ( a ∗ ) ,

Therefore, we have, for all large enough t ≥ τ ,

The variance of noise is, which implies that,

̸

Therefore, we have,

̸

Using Lemma 15 with x n = ∑ t -1 s = τ (1 -π s ( a ∗ )) > 0 , x n +1 = ∑ t s = τ (1 -π s ( a ∗ )) > 0 , c = C > 0 and B = K -1 ≥ 1 gives us that for all t > τ ,

where M = max { K -1 , 1 C ln(( K -1) C )) , (1 -π τ ( a ∗ )) } = K -1 .

Finally, we use Eq. (252) to bound the average sub-optimality. For any s ≥ τ and T > τ ,

̸

Summing from s = τ to T ,

Lemma 13. Let { y n } be the solution to the difference equation y n +1 = y n + Be -cy n with B ≥ 1 , c > 0 and y 0 ≥ max( B, 1 c ln( Bc )) . Let { x n } be a nonnegative valued sequence such that x 0 ≤ y 0 and x n +1 ≤ x n + Be -cx n for all n ≥ 0 . Then, x n ≤ y n for all n ≥ 0 .

Proof. Define the function f ( y ) = max { x + Be -cx : 0 ≤ x ≤ y } . Clearly, f is an increasing function of its argument.

Claim: For y ≥ max( B, 1 c ln( Bc )) , f ( y ) = y + Be -cy . To prove this claim, for x ∈ R , define

Function g is increasing on ( 1 c ln( Bc ) , ∞ ) and decreasing on ( -∞ , 1 c ln( Bc )) .

If c < 1 / B , 1 c ln( Bc ) < 0 , hence g is increasing on (0 , ∞ ) . Hence, f ( y ) = g ( y ) , proving the claim.

If c ≥ 1 / B , then 1 c ln( Bc ) ≥ 0 . Hence, g is decreasing on (0 , 1 / c ln( Bc )) and then increasing on ( 1 / c ln( Bc )) . Since y ≥ 1 c ln( Bc ) , f ( y ) = max( g (0) , g ( y )) . Since we also have y ≥ B > 1 c ln( Bc ) , g ( y ) ≥ g ( B ) > B = g (0) and thus f ( y ) = g ( y ) , finishing the proof of the claim.

The difference equation for y n is y n +1 = y n + Be -cy n . Since y 0 ≥ max( B, 1 c ln( Bc )) and y n is increasing, we have y n ≥ max( B, 1 c ln( Bc )) for all n . Therefore, we can apply the equality from the previous claim to get

The sequence ( x n ) satisfies the inequality x n +1 ≤ x n + Be -cx n ≤ f ( x n ) .

Now, let us prove x n ≤ y n using induction. The base case is x 0 ≤ y 0 (given). Assume x k ≤ y k for some k ≥ 0 . Since f is increasing, x k ≤ y k implies f ( x k ) ≤ f ( y k ) . Using the properties x k +1 ≤ f ( x k ) and f ( y k ) = y k +1 , we get x k +1 ≤ f ( x k ) ≤ f ( y k ) = y k +1 . By the principle of mathematical induction, x n ≤ y n for all n ≥ 0 .

Lemma 14. Let c > 0 and B ≥ 1 , and let { y n } ∞ n =0 be a sequence defined by the recurrence relation

s.t. y 0 ≥ max( B, 1 c ln( Bc )) . Then,

Proof. Define the function:

and note that it is an increasing function, Moreover, ˙ y ( t )(= d dt y ( t )) = e -cy ( t ) for any t ≥ 0 . Hence,

and note that g is increasing when y ≥ 1 c ln( Bc ) . Moreover, y (0) = y 0 and y n +1 = g ( y n ) .

Since y (0) = y 0 ≥ 1 c ln( Bc )) and both { y n } and y ( t ) are increasing, y n ≥ 1 c ln( Bc ) for all n and , y ( t ) ≥ 1 c ln( Bc ) for all t ≥ 0 .

We first prove that

Define the function:

We prove Eq. (254) by induction. The claim holds for n = 0 by construction. Now assume that y ( n ) ≤ y n holds for some n ≥ 0 . Let us show that that y ( n +1) ≤ y n +1 also holds. For this note that

(induction hypothesis, g is increasing for y ≥ 1 c ln( Bc ) and y n , y ( n ) ≥ 1 c ln( Bc ) ) = y n +1 . (By definition of y n +1 )

By the principle of mathematical induction, y ( n ) ≤ y n for all n ≥ 0 .

Define ∆ n := y n -y ( n ) . From our previous inequality, we know that ∆ n ≥ 0 . We now show that { ∆ n } n is bounded, from which the desired statement follows immediately. To show that { ∆ n } n is bounded we will show that ∆ n +1 -∆ n is summable. To show this, we start by obtaining an expression for ∆ n +1 -∆ n . Let A = e cy 0 . Let n ≥ 0 . Direct calculation gives

Using that for all x > 0 , ln(1 + x ) ≥ x 1+ x 2 , we get

For a fixed m> 0 , summing up the above inequality from n = 0 to m -1 ,

where π = 3 . 14159 . . . . Hence, it follows that for any n ≥ 0 ,

Lemma 15. Let { x n } be a nonnegative valued sequence such that x n +1 ≤ x n + Be -cx n for all n ≥ 0 with B ≥ 1 , c > 0 . Then, for all n ≥ 0 ,

where M = max { B, 1 c ln( Bc )) , x 0 } .

Proof. Let { y n } be the solution to the difference equation y n +1 = y n + Be -cy n where y 0 = max { B, 1 c ln( Bc )) , x 0 } . Since y 0 ≥ x 0 and y 0 ≥ max( B, 1 c ln( Bc )) , we can use Lemma 13 to conclude that for all n ≥ 0 ,

Furthermore, using Lemma 14 we can conclude that,

Combining the above inequalities completes the proof.

D Additional simulation results

0.0

0.2

0.4

0.6

0.8

1.0

t

1e6

10

7

10

6

10

5

10

4

10

3

10

2

10

1

10 0

Probability of optimal action

(a)

π

θ

t

(

a

∗

)

,

η

= 100

.

0.0

0.2

0.4

0.6

0.8

1.0

t

1e6

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Sub-optimality gap

(b)

r

(

a

∗

)

-

π

⊤

θ

t

r

,

η

= 100

.

0

2

4

6

8

10

12

14

log(t)

18

16

14

12

10

8

6

4

2

Log sub-optimality gap

(c)

log (

r

(

a

∗

)

-

π

⊤

θ

t

r

)

,

η

= 10

.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In our view, the abstract and introduction summarize the main results, as well as the technical novelty in obtaining these results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We describe the setting and assumptions clearly. The main results are backed by proof sketches in the main text and detailed proofs in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our experiments consist of simple simulations, for which all the details are included in Section 4. The algorithm studied here is remarkably simple and well-known, so the experiments should be straightforward to reproduce with these details, without any code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Given that we are doing simulations with a very classical algorithm, for which we give all simulation details and hyperparameters, we do not believe that code is necessary to reproduce the results. Our data is simulated, and hence easily reproduced, given the distribution which we explicitly describe.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see the details in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: There are no comparisons for which significance needs to be demonstrated, but we provide evidence for the convergence by repeating 10 independent runs, which are presented in the plots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: Our experiments are simply run on a single laptop or Python notebook, without requiring any particular infrastructure.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research does not involve human subjects or introduces any new datasets. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No data or models are released.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: No existing assets are used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No human subjects or crowdsourcing is involved in this work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human subjects or crowdsourcing is involved in this work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies stochastic gradient bandits with arbitrarily large learning rates. The authors prove an interesting result -- the learning rate of the gradient bandit algorithm (in particular, REINFORCE without baseline) can be arbitrarily large. Some numerical simulations are also provided to validate the findings.\n\nSTRENGTHS:\n1. Theoretically, the authors prove that for any constant learning rate, the stochastic gradient bandit algorithm (REINFORCE without baselines) converges to the globally optimal policy almost surely as the number of iterations goes to infinity. The authors also avoid using non-uniform smoothness and noise growth conditions in a previous related work ([1]).\n\n2. The authors also provide some simulation results to verify their theoretical findings.\n\n\n\n[1] Stochastic gradient succeeds for bandits.\n\nWEAKNESSES:\n1. Optimization with large learrning rates have been studied, but the authors seem to only focus on the RL settings. It might be helpful to also include some literature review on deep learning with large learning rates (see, e.g., [1]).\n\n2. Although it is shown that the algorithm can converge with arbitrarily large learning rates, it is still unclear what the exact convergence rate is. In contrast, a previous work [2], as also mentioned by the authors, has the finite-time convergence guarantees.\n\n3. The code is not provided, although the experimental setting is simple.\n\n[1] Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability\n\n[2] Stochastic gradient succeeds for bandits.\n\nQUESTIONS:\n1. Algorithm 1 is the gradient bandit algorithm without baselines. Can we also have similar results for the version with baselines?\n\n2. What would be the challenges of extending the current analysis to the non-asymptotic rate analysis? Furthermore, what would be the challenges of extending the current proof techniques to other RL algorithms like natural gradient methods?\n\nLIMITATIONS:\n1. The convergence results only imply asymptotic convergence without an explicit rate. Thus one direct question one may ask is, how does the learning rate affects the convergence speed. It would be interesting to study if there is an optimal learning rate.\n\n2. The current results are only limited to REINFORCE without baselines for bandit setting, which is simpler than more complex RL settings. Hence it is unclear if it can provide any guidance to RL research in general.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This work studies the asymptotic global convergence rate of the stochastic gradient bandit algorithm with an arbitrary constant learning rate and proves that this algorithm asymptotically converges to the global optimal. This work reveals how this algorithm balances exploitation and exploration and proves the results by contradiction and reduction. This work also provides simulation experiments to support their results.

STRENGTHS:
The proof process based on contradiction and reduction is novel. Furthermore, the analysis reveals why the stochastic gradient bandits algorithm can naturally balance exploitation and exploration, which deepens the understanding of this algorithm.

WEAKNESSES:
The empirical success of a large learning rate is unclear. This work shows that softmax policies and logistic regression can be learned when using a large learning rate. However, they do not show that a large learning rate can lead to great empirical performance.

QUESTIONS:
Question 1: As shown in Figure 1, when choosing $\eta= 100$ or $1000$, the algorithm does not converge. Can you discuss this phenomenon in detail?

Question 2: [1] consider $K=10$ in the simulation study. However, this work does simulation experiments with small arm numbers such as $K=4$ or $K=2$. It would be better to consider larger $K$ and show that the learning rate would not be influenced by $K$.

Comment 1: It would be helpful to discuss the challenge when using the technique in [1] for a constant learning rate.  

[1] Mei, J., Zhong, Z., Dai, B., Agarwal, A., Szepesvari, C., & Schuurmans, D. (2023, July). Stochastic gradient succeeds for bandits. In International Conference on Machine Learning (pp. 24325-24360). PMLR.

LIMITATIONS:
The authors adequately addressed the limitations and societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel theoretical analysis of the stochastic gradient bandit algorithm, showing that it converges to a globally optimal policy almost surely using any constant learning rate. This result is significant as it extends the understanding of stochastic gradient methods in bandit settings, even when standard smoothness and noise control assumptions do not hold.\n\nSTRENGTHS:\n1. Theoretical Contribution: The paper provides a strong theoretical result, proving the global convergence of the stochastic gradient bandit algorithm for any constant learning rate. This is a significant advancement over existing literature that typically requires decaying or small learning rates.\n2. Novel Insights: The authors uncover interesting properties of action sampling rates and the relationship between cumulative progress and noise, contributing to a deeper understanding of exploration-exploitation trade-offs in stochastic gradient methods.\n3. Clarity and Rigor: The proofs are presented with clarity and rigor, making the paper accessible to readers with a solid background in stochastic optimization and reinforcement learning.\n4. Empirical Validation: The simulation studies support the theoretical findings, demonstrating the convergence behavior under various learning rates.\n\nWEAKNESSES:\n1. Practical Implications: While the theoretical results are robust, the practical implications, especially regarding the choice of learning rate in real-world applications, are not thoroughly discussed. It would be beneficial to include more guidance on how practitioners can leverage these findings.\n2. Rate of Convergence: The paper establishes almost sure convergence but does not provide specific rates of convergence. Including more detailed analysis or conjectures about the rate would enhance the practical utility of the results.\n3. Generalization: The study is limited to multi-armed bandits. Extending the results to more general reinforcement learning settings would significantly increase the impact of the work.\n4. Assumption 1: The assumption that the true mean reward has no ties (Assumption 1) is restrictive. Addressing this limitation or providing more discussion on how this assumption might be relaxed in future work would strengthen the paper.\n\nQUESTIONS:\nProof Details:\n\nCould you provide more detailed explanations for Lemma 1 and Lemma 2? Specifically, how do the assumptions ensure the boundedness of parameters?\nExperimental Results:\n\nIn the experimental section, you use different learning rates (η). Could you elaborate on the observed differences in convergence behavior for these learning rates? How does the algorithm perform with learning rates outside the tested range?\nPractical Implications:\n\nHow would you recommend practitioners choose an appropriate learning rate for real-world applications based on your findings? Are there any heuristics or rules of thumb that can be derived from your results?\nGeneralization:\n\nYour results are currently limited to multi-armed bandits. What challenges do you foresee in extending these results to more general reinforcement learning settings? How might the approach change?\nAssumption 1:\n\nAssumption 1 states that the true mean reward has no ties. How critical is this assumption to your results? Do you have any ideas on how to relax this assumption in future work?\nConvergence Rate:\n\nWhile you have shown almost sure convergence, the specific rate of convergence is not detailed. Could you provide more insights or conjectures about the expected rate of convergence for different learning rates?\nLimitations and Future Work:\n\nYou mention that a more refined analysis is needed to explain the subtleties of different stages of convergence. Could you suggest specific directions or methods for this refined analysis?\n\nLIMITATIONS:\nno", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper reveals that the stochastic gradient bandit algorithm converges to a globally optimal policy almost surely using any constant learning rate. This result stands even when traditional smoothness and noise control assumptions are not met, showing the algorithm’s balance between exploration and exploitation.\n\nSTRENGTHS:\n1. The paper provides useful asymptotic insights into stochastic gradient bandits, which to the best of my knowledge are novel.  Unlike previous work, where the methodology would not work for large learning rates,  the gradient bandit algorithm proposed by the authors is proven to asymptotically converge to the globally optimal policy.\n2. The paper discusses how exploration and exploitation trade-offs are balanced without exploration bonuses. Significant insights are provided into how constant learning rates affect the algorithm’s ability to explore different actions and avoid getting stuck in sub-optimal points.\n3. The proof sketches for Theorems 1 and 2 are particularly beneficial in elucidating the intuition underpinning the proofs. The contradiction-based argument presented in the case where ( $K \\geq 2$) is especially neat and clever.\n\nWEAKNESSES:\n1. As the authors also acknowledge, Assumption 1 seems to be a strong one and possibly unrealistic in applications.\n2. The work establishes almost sure convergence to the globally optimal policy as $t \\rightarrow \\infty$, which doesn't tell anything about the rate of convergence.\n3. Section 5.1 in [23] appears to have similar results as the one presented in this paper. In the discussion of the natural exploratory behavior is attributed to the softmax Jacobian in the update, which is also discussed extensively in [23]. However, I understand that the main focus of  [23] is to show that the stochastic gradient bandit algorithm converges to a globally optimal policy at an $O(1/t)$ rate. I am concerned that this paper loses significance as [23] already establishes convergence rates to the globally optimal policy. \n4. Even though the main focus of the paper is theoretical, it would be nice to see how the stochastic gradient compares to other bandit algorithms such as UCB and Thompson Sampling in numerical experiments.\n\n\n*[23] Mei, Jincheng, et al. \"Stochastic gradient succeeds for bandits.\" International Conference on Machine Learning. PMLR, 2023.*\n\nQUESTIONS:\n1. The paper focuses on the simplest setting of the stochastic bandit problem, where decisions only matter for one step. The more common consideration in bandits literature is to maximizing the cumulative reward (or minimizing the expected regret). Can we apply the stochastic gradient bandit algorithm in that setting?\n2. The numerical experiments section in the paper indicates that smaller learning rates perform better during early optimization stages, while larger rates are beneficial later. I have two related thoughts to this and was wondering if any comments could be made regarding the same:\n- For adaptive learning rates, can one think of running the algorithm in batches, such that the learning rates are decreased systematically for consequent batches?\n- Intuitively the right learning rate should depend on the underlying difficulty of the reward generating mechanism for different arms, however I am not sure if that is apparent from the discussion in the paper. \n3. Can this be extended to the contextual bandits framework?\n\nLIMITATIONS:\nYes, limitations have been adequately addressed. Societal impact statement N/A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

comments: SUMMARY:
This paper proved that stochastic gradient bandits converge to a globally optimal policy almost surely for arbitrary constant learning rates if true mean reward has no ties.

STRENGTHS:
This work extends the previous convergence results of stochastic gradient bandits by generalizing from a specific constant learning rate to an arbitrary constant learning rate. The high-level idea is demonstrated, and simple simulations are conducted to validate the theoretical finding.

WEAKNESSES:
This paper heavily relies on prior work [23], which established the asymptotic rate for a constant learning rate. As a follow-up study, however, I believe this paper does not present sufficiently strong results to warrant acceptance at a top conference.

QUESTIONS:
In simulations, if the learning rate is initially set to large number and then decreased, does algorithm show fast convergence?

LIMITATIONS:
Yes, the authors address the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-14