Title: A Simple and Adaptive Learning Rate for FTRL in Online Learning with Minimax Regret of $\Theta(T^{2/3})$ and its Application to Best-of-Both-Worlds
Venue: NeurIPS 2024
Accepted: yes
Keywords: follow-the-regularized-leader

adaptive learning rate

best-of-both-worlds algorithm

partial monitoring

graph bandits

online learning
Abstract: Follow-the-Regularized-Leader (FTRL) is a powerful framework for various online learning problems. By designing its regularizer and learning rate to be adaptive to past observations, FTRL is known to work adaptively to various properties of an underlying environment. However, most existing adaptive learning rates are for online learning problems with a minimax regret of $\Theta(\sqrt{T})$ for the number of rounds $T$, and there are only a few studies on adaptive learning rates for problems with a minimax regret of $\Theta(T^{2/3})$, which include several important problems dealing with indirect feedback. To address this limitation, we establish a new adaptive learning rate framework for problems with a minimax regret of $\Theta(T^{2/3})$. Our learning rate is designed by matching the stability, penalty, and bias terms that naturally appear in regret upper bounds for problems with a minimax regret of $\Theta(T^{2/3})$. As applications of this framework, we consider three major problems with a minimax regret of $\Theta(T^{2/3})$: partial monitoring, graph bandits, and multi-armed bandits with paid observations. We show that FTRL with our learning rate and the Tsallis entropy regularizer improves existing Best-of-Both-Worlds (BOBW) regret upper bounds, which achieve simultaneous optimality in the stochastic and adversarial regimes. The resulting learning rate is surprisingly simple compared to the existing learning rates for BOBW algorithms for problems with a minimax regret of $\Theta(T^{2/3})$.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a new adaptative learning rate framework for hard problems, called Stability-Penalty-Bias matching learning rate (SPB-matching). Using this SPB-matching learning rate, the paper proposes a Best-of-both-worlds (BOBW) algorithm framework for hard online learning problems. It not only achieves simultaneous optimality in the stochastic and adversarial regimes but also achieves the MS-type bound in the adversarial regime with a self-bounding constraint. The paper further shows the utility of their frameworks by studying two hard problems: partial monitoring and graph bandits.\n\nSTRENGTHS:\n- **Clear structure**: The paper is well-organized, starting with the establishment of the framework for the learning rate, followed by the introduction of an algorithmic framework based on this learning rate, and finally exploring two applications of their frameworks.\n- **Relatively comprehensive study**: The paper considers both the stochastic and adversarial regimes, as well as an intermediate regime termed the adversarial regime with a self-bounding constraint. It provides algorithms and theoretical guarantees, along with specific parameter choices in Algorithm 1 for two particular problems.\n\nWEAKNESSES:\n- **Need more background introduction**: The authors should consider including an optimization problem aligned with (1) to better illustrate the hard problems. Additionally, more explanation is needed for the terms $z_t, h_t, u_t$ in the introduction or preliminaries sections to enhance understanding.\n- **Strong assumptions**: Although the assumptions in Theorem 7 are checked for two specific problems, they are quite strong and may not hold for many hard problems. The authors should add more discussion about these assumptions, explaining the meaning of each assumption and identifying scenarios in which these assumptions hold.\n\nQUESTIONS:\n- Can the authors provide more explanations on the terms $z_t$, $h_t$, and $u_t$?\n- On page 5, line 146, the paper assumes access to $\\hat{h}_t$, which upper bounds $h_t$. Why is this always the case in reality?\n- Why does Theorem 7 not depend on $p_0$?\n- Why are $z_t$ and $u_t$ chosen as they are in the two examples? What is the practical meaning behind this choice? How should $z_t$ and $u_t$ be decided for other examples?\n- Typo: In Lemma 5, line 163, it should be $J \\in \\mathbb{N}$ instead of $j \\in \\mathbb{N}$.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to develop a new adaptive learning rate framework for the Follow-the-Regularized-Leader (FTRL) algorithm that addresses online learning problems with a minimax regret of $\\Theta(T^{2/3})$. \n\nIt specifically targets problems with indirect feedback, such as partial monitoring and graph bandits, and demonstrates the efficacy of the proposed framework in improving Best-of-Both-Worlds (BOBW) regret upper bounds.\n\nSTRENGTHS:\n1. **Innovative Approach:** The SPB-matching learning rate is a novel contribution that simplifies the design of adaptive learning rates for complex online learning problems. It has the potential to be applied to other settings.\n2. **Unified Framework:** It successfully unifies the BOBW guarantee for hard problems, representing a good theoretical advancement. Its application to partial monitoring and graph bandits provides tangible improvements over existing methods.\n\nWEAKNESSES:\n1. **Practical Consideration:** The effectiveness of the SPB-matching framework depends on the proper tuning of parameters, which might limit its practical applicability without further optimization techniques. Furthermore, there are no experiments, as a result of which, it is unclear how the proposed method works in practice, even in the simulation. \n2. **Clarity and Accessibility:**  The paper is difficult to follow due to its technical nature. Some equations and results are presented without sufficient explanation, and the extension to the two case studies introduces many new concepts that are hard to follow.\n\nQUESTIONS:\n1. **Extension to High-dimensional Sparse Bandits:** The $\\Theta(T^{2/3})$ regret reminds me of the results in high-dimensional sparse linear bandits [1*]. The regret in [1*] is $\\Theta(s^{1/3}T^{2/3})$ (where $s$ is the level of sparsity), and the algorithm in [1*] also depends on a sort of \"forced exploration.\" Is it possible to apply the framework developed in this paper to this high-dimensional and sparse setting?\n   \n    -  [1*] Hao, Botao, Tor Lattimore, and Mengdi Wang. \"High-dimensional sparse linear bandits.\" Advances in Neural Information Processing Systems 33 (2020): 10753-10763.\n\n2. **Other Upper Bounds:** The introduction of the minimax regret $\\Theta(T^{2/3})$ is somewhat abrupt. It makes me wonder whether in some cases, there is a minimax regret like $\\Theta(T^{a/(a+1)})$ for some parameter $a>0$ and why the case $a=2$ is so special that we should focus on it.\n\n3. **General Lower Bound:** Is there any general lower bound for the regret in Theorem 2? While particular cases may have existing lower bounds, a general tight lower bound is expected given that the paper starts from a general framework and provides a general upper bound.\n   \n\n4. **Clarification on Exploration Rate Requirement:** Could the author explain why $\\gamma_t \\ge u_t/\\beta_t$ (in line 138) is needed? In which lemma or proof is this condition necessary?\n   \n5. **Comparison of SPB-Matching Rules:** Which rule is better in theory or practice: Rule 1 or Rule 2 in (6)?\n   \n6. **High-Level Explanation of Regret Improvement:** Could the author briefly explain why the regret is improved after applying the established framework in Sections 5 and 6? Is there any high-level explanation beyond the technical details?\n\nLIMITATIONS:\nSee the Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose a simple and adaptive learning rate for FTRL, which can achieve the minimax regret of $O(T^{2/3})$ for some hard online learning problems. Specifically, they have applied their algorithm (FTRL with the proposed learning rate) to achieve the best-of-worlds regret bounds for partial monitoring (with global observability) and graph bandits (with weak observability), respectively. For the first problem, compared with the best existing results, this paper achieves an $\\log(T)/\\log(k)$ improvement in the stochastic setting, and an $(\\log(T)+\\log^2(T)/\\log(k))^{1/3}$ improvement in the adversarial setting. But, for the second problem, compared with the best existing results, it seems that this paper can only achieve worse regret bounds (by a factor of $\\delta^\\ast/\\delta$) in the stochastic and adversarial settings.\n\nSTRENGTHS:\nCompared with previous studies, this paper has the following strengths.\n1) A simple and adaptive learning rate is proposed for FTRL with forced exploration, which can achieve the minimax regret for some hard online learning problems, such as partial monitoring (with global observability) and graph bandits (with weak observability).\n2) The authors have improved the best-of-worlds regret bounds for partial monitoring by utilizing the proposed learning rate. Specifically,  compared with the best existing results, this paper achieves an $\\log(T)/\\log(k)$ improvement in the stochastic setting, and an $(\\log(T)+\\log^2(T)/\\log(k))^{1/3}$ improvement in the adversarial setting.\n\nWEAKNESSES:\nHowever, I have some concerns about this paper.\n1) The proposed learning rate is highly inspired by an existing adaptive learning rate in a recent work [26]. Although the existing one can only be utilized to achieve the minimax regret bound of $O(\\sqrt{T})$ for some relatively easy problems, the only difference for the proposed learning rate is to further trade off a bias term (with the stability term), which does not bring enough challenges.\n2) The motivation for such a learning rate for FTRL is to deal with some hard online learning problems (e.g., partial monitoring and graph bandits) that have been well-studied before. Although this paper has applied their algorithm (FTRL with the proposed learning rate) to achieve the best-of-worlds regret bounds for partial monitoring and graph bandits respectively, it does not bring significant advantages compared with existing results, i.e., only logarithmic improvements for partial monitoring and even worse results for graph bandits.\n3) In Section 3, two update rules are proposed for the simple and adaptive learning rate. However, in Sections 4 and 5, it seems that only the second update rule is utilized. So, it is confusing why the authors propose the first update rule and spend non-ignorable spaces in the main text to introduce the corresponding results.\n4) The writing of this paper needs to be improved. For example, the majority of the literature review is only proposed in the appendix, which indeed reduces the readability of this paper. In addition, in line 273, the authors emphasize that \"Our bound is the first BOBW FTRL-based algorithm with the $O(\\log T)$ bound ...\". Although this is not wrong, it is not much to be particular about, because there already exist other non-FTRL-based algorithms to achieve the  $O(\\log T)$ regret bound.\n\nQUESTIONS:\nBesides the concerns discussed above, I also have the following two questions.\n1) Can the authors provide a detailed comparison between the proposed algorithm and existing algorithms in works [26, 54]?\n2) It seems that Dann et al. [15] propose a black-box approach to best-of-worlds regret bounds in bandits. Can their approach be utilized to solve the partial monitoring problem?\n\nLIMITATIONS:\nThe authors have provided some discussions on the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

comments: SUMMARY:
This paper considers the online problems with a minimax regret of $\Theta(T^{2/3})$ and proposes a new learning rate tuning based on the FTRL algorithm to tackle these problems. The development of the learning rate is straightforward and simpler compared to previous works. The proposed algorithm is versatile in various applications, and the authors prove that it can achieve improved best-of-both-worlds regret bounds.

STRENGTHS:
- This paper is technical solid though the discussion for the techniques is easy-to-follow.
- It proposes a simpler and straightforward tuning strategy, which leads to better best-of-both-world theoretical guarantees in partial monitoring and graph bandit problems.
- The applications justify their findings.

WEAKNESSES:
The main technique of designing learning rates so that the stability and bias terms are matched seems to already appear in the cited paper [26], and the development of the method seems to focus on how to solve the additional term introduced by the 'forced exploration' strategy. With a quick glance at the cited paper [26], in Section 5.1, this paper also employs the 'forced exploration' strategy, except that the balance ratio $\gamma$ is fixed. It would be beneficial to further discuss the difficulties encountered when tackling the 'hard problem'.

QUESTIONS:
Could the author respond to the concerns in the 'Weaknesses'?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel adaptive learning rate framework for Follow-the-Regularized-Leader (FTRL) tailored to online learning problems characterized by a minimax regret of Θ(T^2/3). This new learning rate, termed Stability-Penalty-Bias (SPB) matching, is designed by balancing stability, penalty, and bias terms within the decomposition of the regret as $\\text{Reg}\\_T \\\\leq \\\\sum\\_{t=1}^T \\\\frac{z\\_t}{\\\\beta\\_t \\\\gamma\\_t} + \\\\beta\\_1 h\\_1 + \\\\sum\\_{t=2}^T (\\\\beta\\_t - \\\\beta\\_{t-1}) h\\_t + \\\\sum\\_{t=1}^T \\\\gamma\\_t$. The paper demonstrates the efficacy of this approach through its application to two significant “hard” online learning problems: partial monitoring and graph bandits, both of which involve indirect feedback. The proposed framework improves upon existing Best-of-Both-Worlds (BOBW) regret bounds, providing simpler yet effective learning rates compared to existing methods and consequently achieving $O(\\\\log T)$ regret in the stochastic regime and $O(T^{2/3})$ in the adversarial regime.\n\nSTRENGTHS:\nThis work proposes a novel and interesting unified framework for analyzing regret in online learning problems, inspired by previous work with similar methods of regret decomposition.\nThis enables new BOBW guarantees for the problems of partial monitoring with global observability and graph-feedback bandits with weak observability, significantly improving over previous work.\nThe main results are also introduced with sufficient generality that can help the application and adoption of the proposed SPB-matching technique to other “hard” problems as suggested by the authors.\nAdditionally, the same method is shown to retrieve regret bounds under the more general setting with self-bounding constraints.\n\nThe algorithm is simple to understand for readers who have familiarity with the related work. It is a variant of the well-known follow-the-regularized-leader (FTRL) with additional exploration (given the weak observability of the considered problems), negative Tsallis entropy regularizer (with an appropriate tuning of its parameter $\\\\alpha = 1-1/(\\\\log k)$), and careful tuning of the learning rate via the proposed SPB-matching.\nEven considering the simplicity of the main methodology, the observations and ideas behind it are nontrivial and clever.\nThey required some care to have all the pieces fit together to achieve the final results.\n\nA secondary but nontrivial observation made in this work is that the generality of the Tsallis entropy provides an improvement to the regret bounds compared to using only Shannon entropy and log-barrier.\nA similar phenomenon has already been noted in previous work, but to the best of my knowledge, this is the first time this observation has been made for the problems considered here.\n\nFinally, the presentation slowly introduces the SPB-matching technique and the ideas behind it help understand how the algorithm works for providing the desired guarantees.\nThis might also help in adopting similar techniques in other settings, possibly providing new and improved BOBW results in additional “hard” online learning problems.\n\nWEAKNESSES:\nThere is a clear effort by the authors to provide a smooth and clear description of the SPB-matching technique.\nHowever, the large number of parameters made it hard to fully understand how each would exactly influence the analysis of the final regret bound of the algorithm.\nOnly after multiple reads did the full details click and it was clear how all the pieces nicely fit together.\nIn any case, this appears to be caused by the generality of this unified framework and is likely to be unavoidable.\n\nFurthermore, the regret bounds in Corollaries 9 and 11 are not fully clear because of the presence of $\\\\kappa$ and $\\\\kappa’$.\nGiven that all the other parameters have specific values for the two problems considered here, and both $\\\\kappa$ and $\\\\kappa’$ are a function of them, replacing them and thus having an explicit dependence on the actual parameters of the problems could make it easier to understand the regret guarantees in the two corollaries and to compare them with previous related results.\nEven a simple comment on the influence of $\\\\kappa$ and $\\\\kappa’$ to the regret bound for in these two specific cases would help the reader.\n\nQUESTIONS:\n- The possibility of adopting similar ideas to other “hard” online learning problems is suggested in the conclusions. However, while the hardness of the problems considered here is mainly related to the observability of the losses, other problems have different characteristics that make them “hard”. For example, the additional hardness of bandits with switching costs is about the loss being adaptive to the previous action of the learner, leading to an extra cost whenever the learner changes the action to play. On the other hand, SPB-matching is related to the presence of the bias term due to having extra exploration to tackle the limited observability. Do the authors believe a variant of SPB-matching could help with these additional problems? Or do they believe these problems would require different novel ideas? Just some vague intuition about this would be appreciated.\n- What are the lower bounds for partial monitoring with global observability? I think being clearer about this would help the reader understand the quality of your regret bound for this problem.\n\nMinor comments/typos:\n- Line 211: “literature” instead of “litarature”\n- Line 221: l.h.s. of the inline equation should be $\\\\sum\\_{c=1}^k \\\\bigl( G(c,\\\\Phi\\_{cx})\\_{a} - G(c,\\\\Phi\\_{cx})\\_{b} \\\\bigr)$\n- Line 695: duplicate “which is not desirable.”\n- Line 700: “Shannon” instead of “Shanon”\n- Lines 701-702: “dominating” instead of “domination”\n\nLIMITATIONS:\nThe authors address potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}