Title: Does Stochastic Gradient really succeed for bandits?
Venue: NeurIPS 2025
Accepted: yes
Keywords: bandits

policy gradient
Abstract: Recent works of Mei et al. (2023, 2024) have deepened the theoretical understanding of the *Stochastic Gradient Bandit* (SGB) policy, showing that using a constant learning rate guarantees asymptotic convergence to the optimal policy, and that sufficiently *small* learning rates can yield logarithmic regret. However, whether logarithmic regret holds beyond small learning rates remains unclear. In this work, we take a step towards characterizing the regret *regimes* of SGB as a function of its learning rate. For two--armed bandits, we identify a sharp threshold, scaling with the sub-optimality gap $\Delta$, below which SGB achieves *logarithmic* regret on all instances, and above which it can incur *polynomial* regret on some instances. 
This result highlights the necessity of knowing (or estimating) $\Delta$ to ensure logarithmic regret with a constant learning rate.
For general $K$-armed bandits, we further show the learning rate must scale inversely with $K$ to avoid polynomial regret. We introduce novel techniques to derive regret upper bounds for SGB, laying the groundwork for future advances in the theory of gradient-based bandit algorithms.


Peer reviews:
comments: SUMMARY:
The paper revisits the SGB algorithm and provides a sharp characterization of when a constant learning rate yields log regret. For two-armed bandits it establishes a threshold at which the regret transitions from logarithmic to polynomial, while for K-armed bandits it shows the step size must scale with the gap divided by K. Synthetic experiments confirm a clear phase transition that validate the theory.

STRENGTHS AND WEAKNESSES:
The work offers an elegant, elementary analysis that pinpoints the precise learning-rate regimes in which constant-step SGB remains efficient. A new regret decomposition isolates a “post-convergence” term that always stays logarithmic from a “failure” term that explodes if the learning rate is too large. The results are clean and tight for the two-arm case and extend to the multi-arm setting. It is an overall solid theoretical paper.

QUESTIONS:
1. Could an adaptive or decaying step-size schedule dispense with explicit gap knowledge while still attaining logarithmic (or better) regret? This question is especially pertinent for large $K$, where your bounds dictate a very small constant step size.

2. Does the analysis point to practical modifications, such as adding an exploration bonus (e.g., entropy regularization) or incorporating a simple variance-reduction technique, that would allow a larger step size without slipping into the polynomial-regret regime?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-24

comments: SUMMARY:
The paper explores the Stochastic Gradient Bandit policy with a focus on the dependence of the regret on the learning rate $\eta$. Relying on the work of Mei et al, 2023,2024, it exhibits different regret regimes depending on the positioning of $\eta$ as compared to the reward gap $\delta$ through a decomposition of the regret in two terms. Starting with a study of the 2-arms instance (upper bound then lower bound for larger $\eta$), it then studies the case where $K\geq 2$. Deepening our understanding of SGB, I believe that it is a solid theoretical contribution to the field.

STRENGTHS AND WEAKNESSES:
I appreciate the following strengths of the paper:
- it is an interesting contribution to improve the understanding of SGB, a promising area of the bandit literature.
- there is a **very solid** amount of theoretical work to obtain the theorems of the paper, also supported by some synthetic experiments.
- I appreciate the fact that it starts with the 2-arms case to help the understanding before diving in the more complex theorems.

On the weaknesses side, I have the following concern:
- the initial setup of SGB could be more clearly formulated at the beginning of the paper (even though it is mostly a matter of (re)organization).

QUESTIONS:
Could the authors provide a more detailed paragraph that would help to understand the (possible?) link between policy gradient algorithms for RL and SGB?

Is there hope for really practical applications of SGB?

LIMITATIONS:
The paper explores a setup proposed in previous works: in that sense, it is not extremely novel.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers \"stochastic gradient bandits\", a simple instance of the stochastic policy gradient method where the actions are sampled from a softmax policy whose parameters are updated via a specific randomized estimate of the gradient of the value of the policy. This algorithm is noteworthy due to its simplicity. Recent previous work has established that any constant learning rate guarantees convergence to the optimal policy, and a small enough (but \"gap-dependent\") learning is sufficient to get an O(1/t) convergence of the policy (implying a log(t) regret bound). This paper refines these previous results in a number of ways:\n(1) It introduces a novel analysis technique that resembles potential function (or Lyapunov function) techniques used in online learning and beyond to directly analyze the regret, rather than analyzing policy convergence.\n(2) With this tool in place, for the 2-armed case, it establishes that if the constant stepsize is large compared to the size of the gap between the rewards of the two arms then the regret will be polynomial on some instances.\n(3) Also for the 2-armed case, it also establishes that for a constant stepsize that is smaller than the gap, the regret will be logarithmic.\n(4) The paper also shows that for the K-armed setting, the regret will be polynomial if the stepsize is large relative to 1/K.\n(5) Preliminary results are given for upper bounding the regret for the general K-armed setting.\n(6) Experimental results illustrate the findings (and in particular, highlight that the phenomenon exploited in the lower bound proofs are real).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) New, neat analysis ideas that perhaps others can build on\n2) Nice \"characterization\" of how constant stepsize influences regret growth\n3) The paper is quite readable\n\nWeaknesses:\n1) Regret is a weaker concept than policy convergence, though I would not say this is a big problem. Yet, because the paper focuses on regret, it won't have results on policy convergence. Perhaps the price to be paid for the nice analysis\n2) Results for the K-armed case are \"unfinished\"\n\n--------\nSince there is no separate box for various miscellaneous comments, I will use this box for small comments/questions.\n\n1) How important was that the parameters are initialized at zero? (Line 79)\n2) Line 84: E_t must be standing for NOT E[.|H_t] but for E[.|H_{t-1}] from what I see.\n3) Line 107: \"fully characterize [..] except\" sounds funny at best. Consider rephrasing\n4) Displayed eqn, first line after line 134: Why is this not an equality?\n5) Line 187 Theorem 2 and proof: This feels a bit messy. The theorem should be stating something like this? \"For every eta, there exist Delta such that for all epsilon>0 and T large enough XYZ holds\". Correct? The proof sketch is also messy (though eventually what is happening is clear). Perhaps consider rephrasing these. Related to this comment, see also a question below.\n6) Line 202: \"Lemma 6)\" delete \")\".\n7) Line 247, Theorem 3: This should start with \"Fix Delta>0\"? Otherwise, can't I just use Theorem 2 to get this result by making Delta->0?\n8) The histograms reminded me of similar observations made in the UCB-Tuned paper by Audibert et al. who also observe similar bimodal regret distributions. Perhaps it would be interesting to compare/contrast to this.\n\nQUESTIONS:\nWith some simplification, Theorems 1 & 2 suggest that there are two regimes for the regret.\n\nFor eta<Delta, the regret is min(Delta T,log(T)/Delta).\nFor eta>Delta, the regret is min(Delta T, T^{1- (Delta/(2eta))}).\n\nNow, if I fix some eta, compared to this there are those problems where the gap is small. This seems to be the second case above (realm of Theorem 2).\nHere, if Delta goes to zero, the minimum is always Delta T. Hence, for very small values of Delta, Theorem 2 is not relevant. \n\nSo it seems that Theorem 2 says that there is an intermediate range of values of Delta, such that regret is nontrivially large.\n\nI guess what really goes on here is that both theorems are essentially asymptotic (they only hold for T > f(1/Delta) for some increasing function f). This sorta means that Delta needs to be large anyways to use the theorems.\n\nI guess I am struggling to understand a little the implications of these results.\n\nTo formulate this as a question, what is the best way to understand what these results are telling us? Simply put, say, I use eta = 1/sqrt{T} or something like this. When should I worry about the performance of this algorithm (T is fixed)?\n\nLIMITATIONS:\nn.a.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the regret bounds of softmax policy gradient (SPG) for Multi-Armed bandits when using a constant learning rate that depends on the minimum reward gap of an bandit instance. Rather than analyzing the method using techniques from optimization, the authors instead consider the regret decomposition using the SPG update. As a result, the author show that depending how close the learning rate is to the reward gap, SPG either achieves logarithmic or polynomial regret. Additionally, the authors conduct experiments on simple bandit problems to support their theoretical results.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. Provides a throughout analysis for SGD when using constants step-sizes and obtain regret bounds that do not depend on $\\inf_{t \\geq 1} \\mathbb{E}[p_{1, t}]$ which appear frequently in prior work.\n2. The paper is well-written and motivated. The proof and proof sketches in the main paper were digestible.\n3. The empirical results support the theoretical findings. \n\n**Weaknesses**\n1. The results appear to require the rewards to have support $[-1, 1]$ which restricts common distributions such as Gaussian distributions\n2. The results for $K >2$ arms seem to be fixated for constant step-sizes depending on $\\Delta$ and does not seem to have analogous results such as Corollary 1.\n\n**Clarity**\nIt would be good to explicitly mention that the smaller dashed-lines represented the percentiles in Figures 1 & 2.\n**Originality**\nI believe this work offers a novel technique for analyzing SPG methods when using constant step-sizes and introduces different view when considering such problems.\n\nQUESTIONS:\n1. This work mostly focuses on constant step-size since the objective satisfies a condition similar to the strong growth condition [1].\nHowever, is it possible to technique introduce in the paper can be extended for decreasing step-sizes (or a reason why it cannot) for $K > 2$ arms?\n\n2. Just to clarify, is not required to assume that all rewards are distinct, i.e. $\\mu(i) \\neq \\mu(j)$ for $i \\neq j$. This is a common assumption in prior work [2] and it's surprising to me that it's not not needed.\n\n3. In the proof sketch of Lemma 1, why are the rewards deterministic?\n\n\n[1] Mei, J., Zhong, Z., Dai, B., Agarwal, A., Szepesvari, C., & Schuurmans, D. (2023, July). Stochastic gradient succeeds for bandits.\n[2] Mei, J., Dai, B., Agarwal, A., Vaswani, S., Raj, A., Szepesvári, C., & Schuurmans, D. (2024). Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates.\n\nLIMITATIONS:\nYes, the authors sufficiently address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}