Title: Revisiting Follow-the-Perturbed-Leader with Unbounded Perturbations in Bandit Problems
Venue: NeurIPS 2025
Accepted: yes
Keywords: multi-armed bandits

best-of-both-worlds

FTPL

utility model

extreme value theory
Abstract: Follow-the-Regularized-Leader (FTRL) policies have achieved Best-of-Both-Worlds (BOBW) results in various settings through hybrid regularizers, whereas analogous results for Follow-the-Perturbed-Leader (FTPL) remain limited due to inherent analytical challenges. 
    To advance the analytical foundations of FTPL, we revisit classical FTRL-FTPL duality for unbounded perturbations and establish BOBW results for FTPL under a broad family of asymmetric unbounded Fréchet-type perturbations, including hybrid perturbations combining Gumbel-type and Fréchet-type tails.
    These results not only extend the BOBW results of FTPL but also offer new insights into designing alternative FTPL policies competitive with hybrid regularization approaches.
    Motivated by earlier observations in two-armed bandits, we further investigate the connection between the $1/2$-Tsallis entropy and a Fréchet-type perturbation.
    Our numerical observations suggest that it corresponds to a symmetric Fréchet-type perturbation, and based on this, we establish the first BOBW guarantee for symmetric unbounded perturbations in the two-armed setting.
    In contrast, in general multi-armed bandits, we find an instance in which symmetric Fréchet-type perturbations violate the standard condition for BOBW analysis, which is a problem not observed with asymmetric or nonnegative Fréchet-type perturbations. 
    Although this example does not rule out alternative analyses achieving BOBW results, it suggests the limitations of directly applying the relationship observed in two-armed cases to the general case and thus emphasizes the need for further investigation to fully understand the behavior of FTPL in broader settings.


Peer reviews:
comments: SUMMARY:
This paper studies FTPL for MAB. While all existing results either use unbounded symmetric perturbation (e.g., Laplace noise) or non-negative Frechet-type noise. This work is the first to show that BOBW bounds can be achieved even using unbounded but asymmetric perturbation, that is, Frechet distribution on both sides but with different tails.

STRENGTHS AND WEAKNESSES:
Strengths

1. BOBW guarantee of FTPL with asymmetric unbounded perturbation is a new result. It also needs new analysis.
2. Sharp remarks/discussions are presented.

Weaknesses

1. My main concern is the motivation of the significance of the result. "These results not only extend the BOBW results of FTPL but also offer new insights into designing alternative FTPL policies competitive with hybrid regularization approaches." How is asymmetric perturbation exactly connected to hybrid regularier in FTRL? How would this new result let us mimick the effect of hybrid regularizer from FTRL into FTPL?

QUESTIONS:
n/a

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work generalizes the BOBW analysis in previous works [1,2] by showing that FTPL can achieve BOBW results using symmetric unbounded Fréchet-type perturbations. Further, the authors also show that FTPL with the symmetric Pareto distribution with shape 2 is also feasible to achieve the BOBW results, in the 2-arm setting. On the negative side, the authors also demonstrate that in the general MABs with more than 2 arms, FTPL with symmetric Fréchet-type perturbations does not satisfy the usual condition for establishing BOBW results.\n\n[1] Honda et al. Follow-the-Perturbed-Leader achieves best-of-both-worlds for bandit problems. ALT, 2023.\n\n[2] Lee et al. Follow-the-perturbed-leader with Fréchet-type tail distributions: Optimality in adversarial bandits and best-of-both-worlds. COLT, 2024.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. **Motivation and Novelty**: Given the progress of FTRL with hybrid regularizers in various bandit problems, it is of great interest to study FTPL with hybrid perturbations. I think both the techniques and the results in this work are valuable and novel to the online learning literature. \n2. **Writing**: This paper is generally well written.\n\n**Weaknesses**\n\nThough most parts of the writing are clear, I personally feel that the writing may be further improved if some parts of the paper could be further explained or discussed. For instance,\n1. For the term “unbounded perturbations” throughout the paper, I am not fully sure how the authors define it. For me, [1] consider the Fréchet distribution with shape 2 on the support $\\mathbb{R}_{\\ge 0}$, which seems to be “unbounded” to me. However, after a complete reading of this paper,  I feel that the authors seem to denote “unbounded distribution” by meaning that a distribution has support on both the positive and negative parts.\n2. From Line 2 to Line 8, the authors seem to express an opinion that FTRL with a hybrid regularizer (roughly) corresponds to FTPL with perturbation, which has support on both the positive and negative parts. I am wondering whether FTRL with a hybrid regularizer can (roughly) correspond to FTPL with a summation of two independent perturbations that only have supports on the positive parts. If so, shouldn’t it be an easier starting point to investigate the FTPL counterparts of FTRL with hybrid regularizers?\n\nBesides, as the main results in the paper body are a bit heavy, I would like to suggest that the authors include a table summarizing the results of this work. I understand that it might be difficult to fit into the main body. Perhaps including such a table in the appendix might also benefit the readers.\n\nQUESTIONS:\n1. There seems to be a concurrent work [3] which simply uses the i.i.d. Fréchet distribution with shape 2 on each base arm and achieves the BOBW results for $m$-set CMAB. Could the authors briefly discuss whether the results of FTPL with hybrid perturbations can be extended to the CMAB setting? If so, are there any advantages of the method in this work compared to that in [3]?\n\n[3] Zhan et al. Follow-the-Perturbed-Leader Approaches Best-of-Both-Worlds for the m-Set Semi-Bandit Problems. arXiv, 2025.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study Follow-the-Perturbed-Leader (FTPL) policies for a large class of asymmetric distributions and derive Best-of-Both-Worlds (BOBW) regret guarantees, i.e., that are optimal in both the stochastic and adversarial settings. The authors also study FTPL with symmetric distributions, where they show that BOBW guarantees are achievable when the number of arms is $K=2$, but that their analysis do not generalize to $K \\ge 3$.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper is theoretically very solid. All statements in the main text are proven in great detail in the appendices.\n\nI found the paper to be well-written and generally clear. The extensive literature review in Sections 2,3, and Appendix A.1 help greatly with situating this work and understanding its contributions. Section 4 is well organized and it is commendable that the leading constants are tracked in the statements of Theorems 4.3 and 4.4. However, I found the exposition in Section 5 somewhat harder to follow.\n\nIn my opinion, the main contribution of the paper is the derivation of BOBW regret guarantees for FTPL under asymmetric perturbations  (Section 4). The BOBW guarantee for symmetric perturbations in two-armed bandits (Section 5.2) is also a meaningful contribution, and the negative results of Section 5.3 are particularly valuable for future work,  but as the authors point out in Remark 5.7, they only show the failure of a specific analysis technique, which does not entail that FTPL with symmetric frechet-type distributions cannot achieve BOBW guarantees.\n\nAdditionally, I have noticed a few minor typos and/or potentially confusing wording (which do not impact my overall positive assessment):\n- Line 105: `might be enable'\n- Line 135: I believe there is a typo in the definition of the importance-weighted estimator. Shouldn't its $i$-th coordinate be  $\\frac{\\ell_{t,I_t}}{\\mathbb{P}(I_t=i)}\\mathbb{1}[I_t=i]$ ?\n- Line 250: 'let $\\lambda_i$ is the...' -> `let $\\lambda_i$ be the...'. I also personally found the wording \"let $\\lambda_i$ be the $\\sigma_i$-th smallest among $\\lambda_1,\\dots,\\lambda_K$\" slightly confusing, and would recommend something like \"let $\\sigma_i$ be the position of $\\lambda_i$ in the nondecreasing ordering of $\\{\\lambda_1,...\\lambda_K\\}$\"\n- Line 298: 'if exists' -> 'if it exists'\n- Line 349: 'do not occur' -> `that does not occur'\n\nQUESTIONS:\n1. Can you give insights regarding why the stochastic regret guarantee of Corollary 4.2 only holds for alpha=2?\n2. Regarding the implications of the negative result in Section 5.3, as you note in Remark 5.7, your results do not constitute a full impossibility proof. Could you elaborate on what this would imply for the prospect of achieving BOBW guarantees with symmetric perturbations? Does it suggest that an entirely new analytical technique is needed, or that perhaps only a small subset of \"well-behaved\" symmetric distributions might work?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFollow-the-Regularized-Leader (FTRL) and Follow-the-Perturbed-Leader (FTPL) are well-known meta-algorithms for online learning problems that achieve optimal rates in both adversarial and stochastic settings. In some cases, they even attain a best-of-both-worlds (BOBW) guarantee.\n\nFor FTRL, prior work has shown that hybrid regularizers, typically combining negative entropy and Tsallis entropy, can be used to obtain a BOBW guarantee. For FTPL, recent research has demonstrated how to use hybrid perturbations to achieve a similar BOBW result.\n\nThis paper extends that line of work by refining the conditions on the types of perturbations used in FTPL and by analyzing both asymmetric and symmetric unbounded perturbations. Specifically, the authors introduce:\n\nA family of asymmetric perturbations that can achieve BOBW guarantees in the general case.\n\nA family of symmetric perturbations that achieve BOBW guarantees in the special case of two arms.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper deepens our theoretical understanding of FTPL, particularly how to design hybrid perturbations that can mimic the behavior of FTRL with hybrid regularizers.\n\nThe study advances the long-term goal of understanding how FTPL can be made comparable to FTRL in a principled and efficient way.\n\nThere are several technically non-trivial contributions related to the analysis of FTPL, which I consider to be the main strength of the paper.\n\nWeaknesses: \n\nAlthough the paper provides additional insights into how to design FTPL algorithms with BOBW guarantees, prior work has already established that FTPL can achieve BOBW under certain settings.\n\nAs a result, the contribution of this paper feels incremental.\n\nI am unsure whether the contribution is significant enough. For me, it is borderline.\n\nQUESTIONS:\nSee the above.\n\nLine 156, Possible typo?\n\nLIMITATIONS:\nLimitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}