Title: Follow-the-Perturbed-Leader Nearly Achieves Best-of-Both-Worlds for the m-Set Semi-Bandit Problems
Venue: NeurIPS 2025
Accepted: yes
Keywords: multi-armed bandit

combinatorial semi-bandit

m-set semi-bandit

best-of-both-worlds

follow-the-perturbed-leader

Fréchet distribution
Abstract: We consider a common case of the combinatorial semi-bandit problem, the $m$-set semi-bandit, where the learner exactly selects $m$ arms from the total $d$ arms. In the adversarial setting, the best regret bound, known to be $\mathcal{O}(\sqrt{nmd})$ for time horizon $n$, is achieved by the well-known Follow-the-Regularized-Leader (FTRL) policy. However, this requires to explicitly compute the arm-selection probabilities via optimizing problems at each time step and sample according to them. This problem can be avoided by the Follow-the-Perturbed-Leader (FTPL) policy, which simply pulls the $m$ arms that rank among the $m$ smallest (estimated) loss with random perturbation. In this paper, we show that FTPL with a Fr\'echet perturbation also enjoys the near optimal regret bound $\mathcal{O}(\sqrt{nm}(\sqrt{d\log(d)}+m^{5/6}))$ in the adversarial setting and approaches best-of-both-world regret bounds, i.e., achieves a logarithmic regret for the stochastic setting. Moreover, our lower bounds show that the extra factors are unavoidable with our approach; any improvement would require a fundamentally different and more challenging method.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the best-of-both-worlds (BOBW) achievability of the Follow-the-Perturbed-Leader (FTPL) algorithm in the combinatorial semi-bandit setting. The authors demonstrate that FTPL achieves near-optimal regret in adversarial environments and logarithmic regret in stochastic settings. While similar results have been established for FTRL-based approaches, this is the first result of its kind for FTPL. This contribution is particularly valuable, as FTPL does not require solving complex optimization problems, which can be computationally expensive in combinatorial settings.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n\nThis paper presents the first BOBW result for FTPL in combinatorial settings, along with improved adversarial regret bounds. Specifically, the authors achieve a regret of $\\sqrt{nmd\\log d}$ improving upon the previous bound of $m\\sqrt{nd\\log(d/m)}$ established by Neu and Bartók (2016). \nAlthough several FTRL-based approaches have achieved optimal adversarial regret (without logarithmic dependencies), they typically require solving complex optimization problems to compute arm-selection probabilities at each round. In combinatorial settings, these optimization steps become significantly more computationally demanding compared to standard multi-armed bandit problems. In this regard, achieving BOBW guarantees with FTPL, an algorithm that avoids solving such optimization problems, is particularly appealing and makes the contribution of this paper especially valuable.\n\n---\n\n### Weakness\n\nIn the current analysis of the stability term, Lemma 4.1 plays a key role, establishing the foundation for Lemmas 4.2, 4.3, and 4.4, which collectively address the main analytical challenges in studying FTPL. While the paper provides a detailed proof showing that $J_{i,N+k,I}/J_{i,N,I}$ is increasing with respect to $\\lambda_q \\geq 0$ for $q\\not\\in I$, the corresponding decreasing result for $q\\in I$ is not provided. The authors simply state that similar arguments would suffice, but I believe this step is not straightforward and may not follow from analogous reasoning.\n\nThis is because, as noted by the authors in Line 193, in the multi-armed bandit setting, the corresponding terms do not involve $1-F$ factors.\nThis structural difference becomes crucial when considering the derivative with respect to $\\lambda_q$ for $q\\in I$, where the presence of $1-F$ terms introduces additional dependencies that were not present in the multi-armed case and $q\\not\\in I$ case. \nTHerefore, the decreasing property for $q\\in I$ may require a more careful, explicit argument rather than a direct extension of the existing proof.\n\nTo be more precise, fix $s \\in I$.\nThen, the partial derivatives of $J_{i,N,I}$ with respect to $\\lambda_s$ is\n$$ \nJ_{i,N,I}^{s}(\\lambda) := -\\frac{1}{2}\\frac{\\partial J_{i,N,I}(\\lambda)}{\\partial \\lambda_s} =\\int_0^\\infty \\frac{e^{-1/(z+\\lambda_s)^2}}{(z+\\lambda_i)^N(z+\\lambda_s)^3} \\prod_{q \\in I \\setminus \\\\{s\\\\}}(1-F(z+\\lambda_q)) \\prod_{q \\not\\int I} F(z+\\lambda_q)  dz.\n$$\nThen, similarly to Line 499,\n$$\n\\frac{\\partial}{\\partial \\lambda_s} \\frac{J_{i,N+k,I}(\\lambda)}{J_{i,N,I}(\\lambda)} = -2 \\cdot \\frac{J_{i,N+k,I}^s(\\lambda)J_{i,N,I}(\\lambda)-J_{i,N+k,I}(\\lambda)J_{i,N,I}^s(\\lambda)}{J_{i,N,I}(\\lambda)^2}.\n$$\nWhile the exponential term can be absorbed into the $Q$ term (as defined in Line 496) when $s\\not\\in I$, since $F$ is expressed in terms of those exponential terms, the same absorption is not possible under the current definition of $Q$ when $s\\in I$.\nIn this case, the exponential term must be explicitly included in the product involving $1-F$.\nTherefore, it is necessary to carefully reconsider the analysis for the $s\\in I$ case by defining\n$$\nQ(z)=\\frac{1}{(z+\\lambda_i)^N}\\prod_{q \\in I \\setminus \\\\{s\\\\}}(1-F(z+\\lambda_q)) \\prod_{q \\not\\in I} F(z+\\lambda_q).\n$$\nThen, if we follow the same steps, we need to consider the sign of\n$$\n\\frac{e^{-1/(z+\\lambda_s)^2}}{(z+\\lambda_s)^3(1-e^{-1/(z+\\lambda_s)^2})}-\\frac{e^{-1/(w+\\lambda_s)^2}}{(w+\\lambda_s)^3(1-e^{-1/(w+\\lambda_s)^2})}.\n$$\nDeriving such results is not straightforward, and therefore the authors should provide detailed proofs to support these claims.\n\nQUESTIONS:\nRegarding the weaknesses, I would like to point out a recent note by Chen and Honda (2025), which argues that the current proof of Lemma 4.1 is incomplete and contains a technical flaw. Furthermore, their numerical validation suggests that the current argument may not hold as intended. I am curious whether the authors believe the proof can be modified or extended to rigorously establish the desired $\\sqrt{nmd\\log d}$ regret bound.\n\nI would also like to emphasize that I fully acknowledge both works were developed independently, and the positive results presented by Chen and Honda have no influence on my overall evaluation. My assessment is based entirely on the concerns about the proof of Lemma 4.1, which plays a critical role in bounding the stability term and supporting the main theoretical results.\n\n---\nChen, Botao, and Junya Honda. \"Note on Follow-the-Perturbed-Leader in Combinatorial Semi-Bandit Problems.\" arXiv preprint arXiv:2506.12490 (2025).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces and analyzes the Follow-the-Perturbed-Leader (FTPL) policy with Frechet perturbations for the m-set semi-bandit problem, which aims overcome the computational complexities of the traditional Follow-the-Regularized-Leader (FTRL) policies. The main contribution is achieving Best-of-Both-Worlds (BOBW) regret guarantees, making it the first FTPL algorithm to approach this performance in the m-set semi-bandit setting when $m\\leq d/2$. Specifically, it demonstrates a near-optimal regret in the adversarial setting and logarithmic regret in the stochastic setting. The authors also establish a lower bound and indicate that the logarithmic factor $(log(d))$ in the adversarial regret is inherent to their analytical approach, and eliminating it would require a fundamentally different method. Furthermore, the work advances theoretical analysis by extending the standard FTRL framework to m-set semi-bandits, a setting where the convex hull of the action set lacks interior points, thereby simplifying previous proofs. The effectiveness of the proposed FTPL algorithm is also supported by empirical validation against various established baselines.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n1) The FTPL policy achieves optimal BOBW regret in both adversarial and stochastic settings, which is an advantage for uncertain environments.\n\n2) FTPL is computationally efficient as it directly selects $m$ arms with smallest perturbed losses, avoiding complex arm-selection probability computations required by FTRL. It uses geometric resampling for unbiased estimators.\n\n3) The paper extends the standard FTRL analysis framework to m-set semi-bandits where the action set's convex hull lacks interior points, simplifying prior proofs by Honda et al.. It also provides a lower bound showing that the log(d) factor in adversarial regret is inherent to their approach.\n\n4) The paper provides detailed empirical validation by comparing FTPL against five baselines (COMBUCB, Thompson Sampling, EXP2, LogBarrier, and a hybrid FTRL) across both stochastic and adversarial environments, with specified parameters, loss generation, and repetitions.\n\n# Weaknesses\n\n1) The theoretical analysis for the stochastic setting (Theorem 3.2) assumes the uniqueness of the optimal action, which might not hold universally in practice.\n\n2) While its application seems novel, FTPL itself is an existing algorithm. The contribution is an extension and refinement of prior FTRL/FTPL analysis, rather than a new algorithmic approach.\n\n3) The FTPL method's adversarial regret includes an unavoidable $log(d)$ factor, preventing it from strictly matching the minimax optimal bound.\n\n4) The BOBW guarantees are only applicable when $m \\leq d/2$. Extending these results to the $m > d/2$ regime remains an open problem.\n\nQUESTIONS:\n1) Could you provide a more in-depth discussion of the specific technical challenges that prevent the removal of the $log(d)$ factor in the adversarial regret bound? It is only noted that obtaining a tighter bound is not possible through bounding the term-wise ratio; and instead, one must analyze the ratio of the full summations directly, which is substantially more challenging. A clearer understanding of this analytical hurdle would be valuable.\n\n2) What are the specific hurdles encountered when attempting to extend the FTPL's BOBW guarantee to the $m > d/2$ regime. It is indicated that FTRL algorithms have achieved BOBW guarantees in this setting, but whether FTPL can match this remains an open problem.\n\n3) What are the implications if there are more than $m$ arms with zero suboptimality gap? How would the analytical framework (e.g., the regret decomposition for the stochastic setting) break down or degrade if this assumption were relaxed?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the BoBW semi-bandit problem. Utilizing and refining recent technical advancements in FTPL with Frechet-type perturbations, this paper derived an FTPL-based algorithm achiving the $\\tilde{\\mathcal O}(\\sqrt{nmd})$ adversarial regret and $\\mathcal O(\\sum_i \\log n \\Delta_i^{-1})$ stochastic regret. While the regret is slightly sub-optimal in terms of logarithmic factors, FTPL avoids the hard-to-solve global optimization problem common in FTRL.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. The elegance and efficiency of FTPL is for sure a plus.\n2. New analytical tools for FTRL is proposed (i.e., it's not a simple adaptation of recent BoBW FTPL results to semibandits).\n3. Numerical illustrations are given.\n\nWeakness:\n\nThe paper is written in a super technical way and lacks some justifications even for experts in this area. For example, around Line 169 the claim \"which is a stronger result and simplifies the proof compared to those of Honda et al. [2023], Lee et al. [2024],\" is not well-elaborated -- while I can get the intuition why this might be simpler as an expert in FTRL, I am not sure whether my understanding is correct.\n\nThe notations are also too many to be easily interpreted.\n\nAlso, the authors are also strongly encouraged to explain the technical difficulty / insights in proving these results: What leads to this stronger result? Is it because of the FTRL-for-FTPL style analysis? If yes, what do the previous works of Honda et al. and Lee et al. use, and what stopped them from using a similar idea? (Note that I do not mean something like \"Line 169 is simply due to an existing observation\", but rather suggest the authors to be clear about their contributions.)\n\nOverall, I'm leaning towards an accept because of the good results provided as well as the new contribution to the somehow-scarce FTPL technical toolbox; nevertheless, as I always worked on FTRL instead of FTPL, I'm not super certain how significant this contribution is.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduced Follow-the-Perturbed-Leader-type algorithm w/ Fréchet-distributed perturbations for the $m$-set combinatorial semi-bandit problem, where a learner selects exactly $m$ out of $d$ arms per round. It is the first to achieve Best-of-Both-Worlds guarantees in this setting: near-optimal $\\mathcal{O}(\\sqrt{n m d \\log d})$ regret under adversarial rewards and logarithmic $\\mathcal{O}(\\sum_{i:\\Delta_i>0} \\frac{\\log n}{\\Delta_i})$ regret for stochastic bandits. FTPL avoids the computational expense of prior FTRL-based BOBW methods by replacing per-step optimization with simple perturbed loss minimization. The analysis reveals the $\\log d$ factor is unavoidable via their proof techniques, establishing a \"fundamental limit of the *proof strategy*\".\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The high-level idea of using geometric resampling to tackle with the lack of full-information feedback aligns with the intuition in the semi-bandit setting and the regret decomposition in Lemma 3.4 is novel can correct.\n- The implication that FTPL without complex tweaks can achieve the best-of-both-worlds guarantee for semi-bandits is a clear and desirable.\n- The \"lower bound for the proof technique\" part (Lemma C.5) is a interesting contribution, which is seldom seen of its kind in the literature.\n\n### Weaknesses\n\n- The last clause from Line 191 to Line 192 is interesting, but its proof from Line 502 to Line 503 is not clear. To be frank, if the authors consider this part as a trivial argument, please give a more clear explanation. However, as far as I can tell, this part is either highly non-trivial or even **incorrect**.\n  - **I'm willing to reconsider my evaluation if the authors can convince me that this part is not wrong**.\n- Lemma C.5 deserves a more intuitive discussion in the Appendix, as it is a key counterpart of the main result.\n\nQUESTIONS:\n- This paper is self-contained, especially regarding technical details. It is also desirable to have a self-contained high-level illustration of the intuition behind the \"Fréchet distribution with shape parameter 2\" that enables the BOBW guarantee, right?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}