Title: Stability-penalty-adaptive follow-the-regularized-leader: Sparsity, game-dependency, and best-of-both-worlds
Venue: NeurIPS 2023
Accepted: yes
Keywords: follow-the-regularized-leader

adaptive learning rate

multi-armed bandits

partial monitoring

data-dependent bound

sparsity

game-dependency

best-of-both-worlds
Abstract: Adaptivity to the difficulties of a problem is a key property in sequential decision-making problems to broaden the applicability of algorithms. Follow-the-regularized-leader (FTRL) has recently emerged as one of the most promising approaches for obtaining various types of adaptivity in bandit problems. Aiming to further generalize this adaptivity, we develop a generic adaptive learning rate, called stability-penalty-adaptive (SPA) learning rate for FTRL. This learning rate yields a regret bound jointly depending on stability and penalty of the algorithm, into which the regret of FTRL is typically decomposed. With this result, we establish several algorithms with three types of adaptivity: sparsity, game-dependency, and best-of-both-worlds (BOBW). Despite the fact that sparsity appears frequently in real problems, existing sparse multi-armed bandit algorithms with $k$-arms assume that the sparsity level $s \leq k$ is known in advance, which is often not the case in real-world scenarios. To address this issue, we first establish $s$-agnostic algorithms with regret bounds of $\tilde{O}(\sqrt{sT})$ in the adversarial regime for $T$ rounds, which matches the existing lower bound up to a logarithmic factor. Meanwhile, BOBW algorithms aim to achieve a near-optimal regret in both the stochastic and adversarial regimes. Leveraging the SPA learning rate and the technique for $s$-agnostic algorithms combined with a new analysis to bound the variation in FTRL output in response to changes in a regularizer, we establish the first BOBW algorithm with a sparsity-dependent bound. Additionally, we explore partial monitoring and demonstrate that the proposed SPA learning rate framework allows us to achieve a game-dependent bound and the BOBW simultaneously.

Stability-penalty-adaptive follow-the-regularized-leader: Sparsity, game-dependency, and best-of-both-worlds

Abstract

Adaptivity to the difficulties of a problem is a key property in sequential decision-making problems to broaden the applicability of algorithms. Followthe-regularized-leader (FTRL) has recently emerged as one of the most promising approaches for obtaining various types of adaptivity in bandit problems. Aiming to further generalize this adaptivity, we develop a generic adaptive learning rate, called stability-penalty-adaptive (SPA) learning rate for FTRL. This learning rate yields a regret bound jointly depending on stability and penalty of the algorithm, into which the regret of FTRL is typically decomposed. With this result, we establish several algorithms with three types of adaptivity: sparsity , game-dependency , and best-of-both-worlds (BOBW). Despite the fact that sparsity appears frequently in real problems, existing sparse multi-armed bandit algorithms with k -arms assume that the sparsity level s ≤ k is known in advance, which is often not the case in real-world scenarios. To address this issue, we first establish s -agnostic algorithms with regret bounds of ˜ O ( √ sT ) in the adversarial regime for T rounds, which matches the existing lower bound up to a logarithmic factor. Meanwhile, BOBW algorithms aim to achieve a near-optimal regret in both the stochastic and adversarial regimes. Leveraging the SPA learning rate and the technique for s -agnostic algorithms combined with a new analysis to bound the variation in FTRL output in response to changes in a regularizer, we establish the first BOBW algorithm with a sparsity-dependent bound. Additionally, we explore partial monitoring and demonstrate that the proposed SPA learning rate framework allows us to achieve a game-dependent bound and the BOBW simultaneously.

1 Introduction

This study considers the multi-armed bandits (MAB) and partial monitoring (PM). In the MAB problem, the learner selects one of k arms, and the adversary simultaneously determines the loss of each arm, ℓ t = ( ℓ t 1 , . . . , ℓ tk ) ⊤ in [0 , 1] k or [ -1 , 1] k . After that, the learner observes only the loss for the chosen arm. The learner's goal is to minimize the regret, which is the difference between the learner's total loss and the total loss of an optimal arm fixed in hindsight. PM is a generalization of MAB, and the learner observes feedback symbols instead of the losses.

Shinji Ito

One of the most promising frameworks for MABs and PM is follow-the-regularized-leader (FTRL) [5, 12], which determines the arm selection probability at each round by minimizing the sum of the cumulative (estimated) losses so far plus a convex regularizer. Note that the well-known Exp3 algorithm developed in [5] is equivalent to FTRL with the (negative) Shannon entropy regularizer. FTRL is also known to perform well for the classic expert problem [17] and reinforcement learning [52]. Furthermore, when the problem is 'benign', it is known that FTRL can exploit the underlying structure to adaptively improve its performance. Typical examples of such adaptive improvements are (i) data-dependent bounds and (ii) best-of-both-worlds (BOBW).

Data-dependent bounds have been investigated to enhance the adaptivity of algorithms to a given structure of losses in the adversarial regime , where feedback ( e.g., losses in MABs) is decided in an arbitrary manner. There are various examples of data-dependent bounds, and this study considers sparsity-dependent bounds and game-dependent bounds .

̸

Asparsity-dependent bound is an important example of data-dependent bounds, as sparsity frequently appears in real-world problems. For example, in online advertisement allocation, it is often the case that only a fraction of the ads is clicked. Although there are some studies for sparse MABs [10, 27, 51], all of them assume that (an upper bound of) sparsity level s ≥ ∥ ℓ t ∥ 0 = |{ i ∈ [ k ] : ℓ ti = 0 }| is known beforehand, which in many practical scenarios does not hold.

The concept of a game-dependent bound was recently introduced by Lattimore and Szepesvári [32] to derive a regret upper bound that depends on the game the learner is facing. As the authors suggest, one of the motivations for the game-dependent bound is that previous PM algorithms are 'quite conservative and not practical for normal problems'. For example, whereas the Bernoulli MAB is expressed as a PM, algorithms for PM do not always achieve the minimax regret of MAB [5]. The game-dependent bound enables the learner to automatically adapt to the essential difficulty of the game the algorithm is actually facing.

The BOBW algorithm aims to achieve near-optimal regret bounds in stochastic and adversarial regimes, where the feedback is stochastically generated in the stochastic regime. Since we often do not know the underlying regime, it is desirable for an algorithm to simultaneously obtain a near-optimal performance both for the stochastic and adversarial regimes. For multi-armed bandits, Bubeck and Slivkins [9] developed the first BOBW algorithm, and Zimmert and Seldin [53] proposed the well-known Tsallis-INF algorithm, which achieves the optimal regret for both regimes. The Tsallis-INF algorithm also achieves favorable regret guarantees in the adversarial regime with a self-bounding constraint , which interpolates between the stochastic and adversarial regimes.

To realize the aforementioned adaptivity in FTRL, the adaptive learning rate (a.k.a. time-varying learning rate) is one of the most representative approaches. This approach adjusts the learning rate based on previous observations. In the literature, adaptive learning rates have been designed to depend on stability or penalty , which are components of a regret upper bound of FTRL. The stability term increases if the variation of FTRL outputs in the adjacent rounds is large, and stability-dependent learning rates have been used in a considerable number of algorithms available in the literature, e.g., [32, 37, 38] and references therein. In contrast, the penalty term comes from the strength of the regularization, and recently penalty-dependent learning rates were considered to achieve BOBW guarantees [22, 46]. However, existing stability-dependent (resp. penalty-dependent) learning rates are designed with the worst-case penalty (resp. stability), which could potentially limit the adaptivity and performance of FTRL. (There are numerous studies related to this paper and we include additional related work in Appendix C.)

1.1 Contribution of this study

In this paper, in order to further broaden the applicability of FTRL, we establish a generic framework for designing an adaptive learning rate that depends on both the stability and penalty components simultaneously, which we call a stability-penalty-adaptive (SPA) learning rate (Definition 2). This enables us to bound the regret approximately by ˜ O ( √ ∑ T t =1 z t h t +1 ) for stability component ( z t ) t and a penalty component ( h t ) t , which we call a SPA regret bound (Theorem 1). With appropriate selections of z t and h t , this result yields the three important adaptive bounds mentioned earlier, namely sparsity, game-dependency, and BOBW. In particular, our contributions are as follows (see also Tables 1 and 2):

(Section 5.1) We initially provide new algorithms for sparse MABs as preliminaries for establishing a BOBW algorithm with a sparsity-dependent bound. In Section 5.1.1, we propose a novel estimator of the sparsity level, which is linked to a stability component and induces L 2 = ∑ T t =1 ∥ ℓ t ∥ 2 ≤ sT . We demonstrate that a learning rate using this estimator with the Shannon entropy regularizer and ˜ Θ(( kT ) -2 / 3 ) uniform exploration immediately results in an O ( √ L 2 log k ) regret bound for ℓ t ∈ [0 , 1] k . In Section 5.1.2, we investigate possibly negative losses ℓ t ∈ [ -1 , 1] k . We employ the time-invariant log-barrier proposed in [10] to control the stability term. This allows us to achieve an O ( √ L 2 log k ) regret bound for losses in [ -1 , 1] k even without the ˜ Θ(( kT ) -2 / 3 ) uniform exploration. This is a key component for developing the BOBW guarantee that we discuss next. Note that Section 5.1 serves as preliminary findings for the subsequent section.

(Section 5.2) We establish a BOBW algorithm with a sparsity-dependent bound. In order to achieve this goal, we make another major technical development: we analyze the variation in the FTRL output when the regularizer changes (Lemma 7), which holds thanks to the time-invariant log-barrier and may be of independent interest. This analysis is necessary since we use a timevarying learning rate to obtain a BOBW guarantee, whereas Bubeck et al. [10] uses a constant learning rate. This technical development successfully allows us to achieve the goal (Theorem 4) in combination with the SPA learning rate developed in Section 4 and a technique for exploiting sparsity in Section 5.1.2.

(Section 6) We show that the SPA learning rate established in Section 4 can also be used to achieve a game-dependent bound and a BOBW guarantee simultaneously, which further highlights the usefulness of the SPA learning rate.

2 Setup

This section introduces the preliminaries of this study. Sections 2.1 and 2.2 formulate the MAB and PM problems, respectively, and Section 2.3 defines regimes considered in this paper.

Notation Let ∥ x ∥ , ∥ x ∥ 1 , and ∥ x ∥ ∞ be the Euclidian, ℓ 1 -, and ℓ ∞ -norms for a vector x , respectively. Let ∥ x ∥ 0 be the number of non-zero elements for a vector x . Let P k = { p ∈ [0 , 1] k : ∥ p ∥ 1 = 1 } be the ( k -1) -dimensional probability simplex. A vector e i ∈ { 0 , 1 } k is the i -th standard basis of R k , and 1 is the all-one vector. Let D Φ be the Bregman divergence induced by differentiable convex function Φ , i.e., D Φ ( p, q ) = Φ( p ) -Φ( q ) -⟨∇ Φ( q ) , p -q ⟩ . Table 3 in Appendix A summarizes the notation used in this paper.

2.1 Multi-armed bandits

In MAB with k -arms, at each round t ∈ [ T ] := { 1 , 2 , . . . , T } , the environment determines the loss vector ℓ t = ( ℓ t 1 , ℓ t 2 , . . . , ℓ tk ) ⊤ in [0 , 1] k or [ -1 , 1] k , and the learner simultaneously chooses an arm A t ∈ [ k ] without knowing ℓ t . After that, the learner observes only the loss ℓ tA t for the chosen arm. The performance of the learner is evaluated by the regret Reg T , which is the difference between the cumulative loss of the learner and of the single optimal arm, that is, a ∗ = arg min a ∈ [ k ] E [ ∑ T t =1 ℓ ta ] and Reg T = E [ ∑ T t =1 ( ℓ tA t -ℓ ta ∗ ) ] , where the expectation is taken with respect to the internal randomness of the algorithm and the randomness of the loss vectors ( ℓ t ) T t =1 .

2.2 Partial monitoring

Formulation A PM game G = ( L , Φ) with k -actions and d -outcomes is defined by a pair of a loss matrix L ∈ [0 , 1] k × d and feedback matrix Φ ∈ Σ k × d , where Σ is a set of feedback symbols. The game is played in a sequential manner by a learner and an opponent across T rounds. The learner begins the game with knowledge of L and Φ . For every round t ∈ [ T ] , the opponent selects an outcome x t ∈ [ d ] , and the learner simultaneously chooses an action A t ∈ [ k ] . Then the learner suffers an unobserved loss L A t x t and receives only a feedback symbol σ t = Φ A t x t , where L ax is the ( a, x ) -th element of L . The learner's performance in the game is evaluated by the regret Reg T as in the MAB case: a ∗ = arg min a ∈ [ k ] E [ ∑ T t =1 L ax t ] and Reg T = E [ ∑ T t =1 ( L A t x t -L a ∗ x t ) ] = E [ ∑ T t =1 ⟨ ℓ A t -ℓ a ∗ , e x t ⟩ ] , where ℓ a ∈ R d is the a -th row of L .

Several concepts in PM Let m ≤ | Σ | be the maximum number of distinct symbols in a single row of Φ ∈ Σ k × d . Different actions a and b are duplicate if ℓ a = ℓ b . We can decompose possible distributions of d outcomes in P d based on the loss matrix. For every action a ∈ [ k ] , cell C a = { u ∈ P d : max b ∈ [ k ] ( ℓ a -ℓ b ) ⊤ u ≤ 0 } is the set of probability vectors in P d for which action a is optimal. Each cell is a closed convex polytope.

Define dim( C a ) as the dimension of the affine hull of C a . Action a is said to be dominated if C a = ∅ . For non-dominated actions, action a is said to be Pareto optimal if dim( C a ) = d -1 , and degenerate if dim( C a ) < d -1 . Let Π be the set of Pareto optimal actions. Two Pareto optimal actions a, b ∈ Π are called neighbors if dim( C a ∩ C b ) = d -2 , which is used to define the difficulty of PM games. A PM game is said to be non-degenerate if it has no degenerate actions. We assume that PM game G is non-degenerate and contains no duplicate actions.

The difficulty of PM games is characterized by the following observability conditions. Neighbouring actions a and b are locally observable if there exists w ab : [ k ] × Σ → R such that w ab ( c, σ ) = 0 for c ̸∈ { a, b } and ∑ k c =1 w ab ( c, Φ cx ) = L ax -L bx for all x ∈ [ d ] . A PM game is locally observable if all neighboring actions are locally observable, and this study considers locally observable games.

Loss difference estimation Let H be the set of all functions from [ k ] × Σ to R d . For any locally observable games, there exists G ∈ H such that for any b, c ∈ Π , ∑ k a =1 ( G ( a, Φ ax ) b -G ( a, Φ ax ) c ) = L bx -L cx for all x ∈ [ d ] [32]. For example, we can take G = G 0 defined by G 0 ( a, σ ) b = ∑ e ∈ path T ( b ) w e ( a, σ ) for a ∈ Π , where T is a tree over Π induced by neighborhood relations and path T ( b ) is the set of edges from b ∈ Π to an arbitrarily chosen root c ∈ Π on T [32]. See Appendix C and [31, Chapter 37] for a more detailed explanation and background of PM.

2.3 Considered regimes

̸

We consider three regimes on the assumptions for losses in MABs and outcomes in PM. In the stochastic regime , a sequence of loss vector ( ℓ t ) in MAB and that of outcome vector ( x t ) in PM follow an unknown distribution ν ∗ in an i.i.d. manner. Define the minimum suboptimality gap in ∆ min = min a = a ∗ ∆ a for ∆ a = E ℓ t ∼ ν ∗ [ ( ℓ ta -ℓ ta ∗ ) ] in MAB and ∆ a = E x t ∼ ν ∗ [ ( ℓ a -ℓ a ∗ ) ⊤ e x t ] in PM. Note that the definitions of ℓ in MAB and PM are different.

In contrast, the adversarial regime does not assume any stochastic structure for the losses or outcomes, and they can be chosen in an arbitrary manner. In this regime, the environment can choose ℓ t for MABand x t for PM depending on the past history until the ( t -1) -th round, ( A s ) t -1 s =1 .

We also consider, the adversarial regime with a self-bounding constraint [53], an intermediate regime between the stochastic and adversarial regimes.

Definition 1. Let ∆ ∈ [0 , 2] k and C ≥ 0 . The environment is in an adversarial regime with a (∆ , C, T ) self-bounding constraint if it holds for any algorithm that Reg T ≥ E [∑ T t =1 ∆ A t -C ] .

One can see that the stochastic and adversarial regimes are indeed instances of this regime, and that well-known stochastic regimes with adversarial corruptions [36] are also in this regime (see [53] and [46] for definitions in MAB and PM, respectively).

We assume that there exists a unique optimal arm (or action) a ∗ , which was employed by many studies aiming at developing BOBW algorithms [18, 34, 49, 53].

3 Preliminaries

This section provides preliminaries for developing and analyzing algorithms. We first introduce FTRL, upon which we develop our algorithms, and then describe the self-bounding technique, which is a common technique for proving a BOBW guarantee.

Follow-the-regularized-leader In the FTRL framework, an arm selection probability p t ∈ P k at round t is given by

where ̂ y t ∈ R k is an estimator of loss ℓ t at round t , Φ t : P k → R is a strongly-convex regularizer, and T t : P k → P k is a map from the output of FTRL q t to an arm selection probability vector p t .

In the analysis of FTRL, it is common to evaluate ∑ T t =1 ⟨ ̂ y t , p t -p ⟩ = ∑ T t =1 ⟨ ̂ y t , q t -p ⟩ + ∑ T t =1 ⟨ ̂ y t , p t -q t ⟩ for some p ∈ P k . It is known (see e.g., [31, Exercise 28.12]) that quantity ∑ T t =1 ⟨ ̂ y t , q t -p ⟩ is bounded from above by

We refer to the terms in (2) as a penalty and stability terms, and to the quantity ⟨ ̂ y t , p t -q t ⟩ as a transformation term. Note that, though this study focuses on examples in which Φ T +1 ( p ) is not dominant, this term may be dominant dependent on the choice of regularizers.

Self-bounding technique A self-bounding technique is a common method for proving a BOBW guarantee [18, 49, 53]. In the self-bounding technique, we first derive regret upper and lower bounds in terms of a variable dependent on the arm selection probabilities ( p t ) t or the FTRL outputs ( q t ) t , and then derive a regret bound by combining the upper and lower bounds. We use a version proposed in [22]. We consider Q ( i ) , ¯ Q ( i ) , P ( i ) , and ¯ P ( i ) for i ∈ [ k ] defined by Q ( i ) = ∑ T t =1 (1 -q ti ) , ¯ Q ( i ) = E [ Q ( i )] , P ( i ) = ∑ T t =1 (1 -p ti ) , and ¯ P ( i ) = E [ P ( i )] . Note that ¯ Q ( i ) , ¯ P ( i ) ∈ [0 , T ] for any i ∈ [ k ] . In terms of ¯ Q ( i ) or ¯ P ( i ) , we can obtain the lower bound of the regret for the adversarial regime with a self-bounding constraint as follows:

Lemma 1 ([46, Lemma 4]) . In the adversarial regime with a self-bounding constraint (Definition 1), if there exists c ′ ∈ (0 , 1] such that p ti ≥ c ′ q ti for all t ∈ [ T ] and i ∈ [ k ] , then Reg T ≥ ∆ min ¯ P ( a ∗ ) -C ≥ c ′ ∆ min ¯ Q ( a ∗ ) -C .

It is known that the sums of the entropy H ( · ) of ( p t ) t is bounded by P ( i ) as follows:

Lemma 2 ([22, Lemma 4]) . Let ( q t ) T t =1 be any sequence of probability vectors and define Q ( i ) = ∑ T t =1 (1 -q ti ) . Then for any i ∈ [ k ] , ∑ T t =1 H ( q t ) ≤ Q ( i ) log(e kT/Q ( i )) .

Based on Lemmas 1 and 2, it suffices to show Reg T ≲ E [√ ∑ T t =1 H ( q t ) polylog( T ) ] to prove a BOBW guarantee in MAB. This is because, for the adversarial regime, using H ( q t ) ≤ log k immediately implies a ˜ O ( √ T ) bound, and for the stochastic regime, using Lemmas 1 and 2 roughly bounds the regret as Reg T = 2 Reg T -Reg T ≲ √ ¯ Q ( a ∗ ) polylog( T ) -∆ min ¯ Q ( a ∗ ) ≲ polylog( T ) / ∆ min .

4 Stability-penalty-adaptive (SPA) learning rate and regret bound

This section proposes a new adaptive learning rate, which yields a regret upper bound dependent on both the stability component z t and penalty component h t for various choices of z t and h t . When we use a learning rate η t , the analysis of FTRL boils down to the evaluation of

In particular, when we use the Exp3 algorithm, h t is the Shannon entropy of the FTRL output at round t . This can be confirmed by checking the existing studies ( e.g., [22, 46]) or the proofs in Appendices F, G, H.2, and I. To favorably bound ̂ Reg SP T , we develop a new learning rate framework, which we call the jointly stability- and penalty-adaptive learning rate, or the stability-penalty-adaptive (SPA) learning rate for short:

Definition 2 (Stability-penalty-adaptive learning rate) . Let (( h t , z t , ¯ z t )) T t =1 be non-negative reals such that h 1 ≥ h t for all t ∈ [ T ] , (¯ z t h 1 + ∑ t s =1 z s h s +1 ) T t =1 is non-decreasing, and ¯ z t h 1 ≥ z t h t +1 for all t ∈ [ T ] . Let c 1 , c 2 > 0 . Then, a sequence of ( η t ) T t =1 is a SPA learning rate if it has a form of

Remark. To the best of our knowledge, this is the first learning rate that depends on both the stability and penalty components. Note that when we set the penalties to their worst-case value, that is, h t = h 1 for all t ∈ [ T ] (recalling h t ≤ h 1 ), the SPA learning rate in (4) becomes equivalent to the standard type of the learning rate, which depends only on the stability and has the form of β t = 1 /η t ≃ c 1 √ h 1 √ ¯ z 1 + ∑ t -1 s =1 z s . On the other hand, when we set the stabilities to be their worst-case value, that is, z ≥ max t ∈ [ T ] z t , the SPA learning rate in (4) corresponds to the learning rate dependent only on the penalty in [22, 46].

Using learning rate ( η t ) in (4), we can bound Reg SP as follows.

̂ T Theorem 1 (Stability-penalty-adaptive regret bound) . Let ( η t ) T t =1 be a SPA learning rate in Definition 2. Then ̂ Reg SP T in (3) is bounded as follows:

(I) If (( h t , z t , ¯ z t )) T t =1 in ( η t ) satisfies √ c 2 +¯ z t h 1 c 1 ( β 1 + β t ) ≥ ε + z t for all t ∈ [ T ] for some ε > 0 (stability condition (S1) ), then

(II) If h t = h 1 for all t ∈ [ T ] , c 2 = 0 , and (( h t , z t , ¯ z t )) T t =1 in ( η t ) satisfies β t ≥ ac 1 √ h 1 √ ∑ t s =1 z s for some a > 0 (stability condition (S2) ), then ̂ Reg SP T ≤ 2 ( c 1 + λ ac 1 ) √ h 1 ∑ T t =1 z t .

The proof of Theorem 1 can be found in Appendix D. In Part (I) of Theorem 1, we can see that ̂ Reg SP T is bounded by √ ∑ T t =1 z t h t +1 , which will enable us to obtain BOBW and data-dependent bounds simultaneously. Note that ̂ Reg SP T , which is the component of the regret, often becomes dominant in particular when we use the Shanon entropy regularizer. Thus, checking the stability condition (S1) and applying Theorem 1 to bound ̂ Reg SP T almost complete the regret analysis. In Section 6, we will see that applying Theorem 1 immediately provides a BOBW bound with a game-dependent bound for PM. In contrast, when deriving BOBW with a sparsity-dependent bound for MAB in Section 5, we will develop additional techniques and conduct further analysis, for example, to satisfy the stability condition (S1), making use of the time-invariant log-barrier regularizer.

5 Sparsity-dependent bounds in multi-armed bandits

This section establishes several sparsity-dependent bounds. We use the FTRL framework in (1) with the inverse weighted estimator ̂ y t ∈ R k given by ̂ y ti = ℓ ti ✶ [ A t = i ] /p ti . This estimator is common in the literature and is useful for its unbiasedness, i.e., E A t ∼ p t [ ̂ y t | p t ] = ℓ t . We first propose algorithms that achieve sparsity-dependent bounds using stability-dependent learning rates in Section 5.1 as preliminaries for the subsequent section. Following that, in Section 5.2, we establish a BOBW algorithm with a sparsity-dependent bound based on the SPA learning rate. More specific steps are summarized as follows.

Section 5.1.1 discusses the case of ℓ t ∈ [0 , 1] k and shows that appropriately choosing z t in the SPA learning rate (4) with the Shannon entropy regularizer and ˜ Θ(( kT ) -2 / 3 ) uniform exploration achieves a O ( √ L 2 log k ) regret for ℓ t ∈ [0 , 1] k without knowing L 2 .

Section 5.1.2 considers the case of ℓ t ∈ [ -1 , 1] k , which is known to be more challenging than ℓ t ∈ [0 , 1] k . We show that the time-invariant log-barrier enables us to choose a 'tighter' z t in (4), which removes the uniform exploration used in Section 5.1.1. This not only results in the bound of O ( √ L 2 log k ) for ℓ t ∈ [ -1 , 1] k but also becomes one of the key properties to achieve BOBW.

Section 5.2 presents a BOBW algorithm with a sparsity-dependent bound using the technique developed in Section 5.1 and Theorem 1. While Theorem 1 itself is a strong tool leading directly to the result for PM (Section 6), its application does not lead to the desired bounds. In particular, in this setting the ˜ O ( √ ∑ T t =1 z t h t +1 ) term derived through Theorem 1 does not immediately imply a BOBW guarantee with a sparsity-dependent bound. To solve this problem, we develop a novel technique to analyze the variation in FTRL outputs q t in response to the change in a regularizer (Lemma 7) , and prove a BOBW bound with a sparsity-dependent bound of O ( √ L 2 log k log T ) .

5.1 Parameter-agnostic sparsity-dependent bounds

This section establishes s -agnostic algorithms to achieve sparsity-dependent bounds for the adversarial regime, which are preliminaries for Section 5.2.

5.1.1 L 2 -agnostic algorithm with O ( √ L 2 log k ) bound for ℓ t ∈ [0 , 1] k

Here, we use p t = T t ( q t ) for T t ( q ) = (1 -γ ) q + γ k 1 and γ = k 1 / 3 (log k ) 1 / 3 T 2 / 3 and assume γ ∈ [0 , 1 / 2] (this holds when T ≥ √ 8 k log k ), which implies 2 p t ≥ q t . We use the Shannon entropy regularizer Φ t ( p ) = -1 η t H ( p ) = 1 η t ψ nS ( p ) = 1 η t ∑ k i =1 p i log p i with learning rate η t = 1 /β t and

which corresponds to the learning rate in Definition 2 with h t ← H ( q 1 ) = log k , z t ← ω t , ¯ z t ← k/γ , and c 2 ← 0 . The uniform exploration is used to satisfy stability condition (S2) in Theorem 1, the amount of which is determined by balancing the regret coming from the uniform exploration and stability condition (S2). Theorem 1 immediately gives the following bound.

Corollary 2. When T ≥ √ 8 k log k , the above algorithm with c 1 = 1 / √ 2 achieves Reg T ≤ 2 √ 2 √ L 2 log k +(2 √ 2 + 1)( kT log k ) 1 / 3 without knowing L 2 . In particular, when T ≥ 7 k 2 /s 3 , Reg T ≤ (4 √ 2 + 1) √ sT log k .

The proof is given in Appendix F. The most striking feature of the algorithm is its L 2 (or s )-agnostic property. This is essentially made possible by the learning rate using the data-dependent quantity ω t in (6), which satisfies E [√ ∑ T t =1 ω t ] ≤ √ ∑ T t =1 E [ ω t ] = √ ∑ T t =1 ∑ k i =1 ℓ 2 ti = √ L 2 . The leading constant of the bound is better than the existing bounds, as shown in Table 1, despite its agnostic property. Note that while the first-order bound in [49] implies the above sparsity-dependent bound when ℓ t ∈ [0 , 1] k , this does not hold when ℓ t ∈ [ -1 , 1] , which we investigate in the following (see Appendix K for details). We will see in Section 5.1.2 that this assumption can be totally removed by adding a time-invariant log-barrier regularization.

5.1.2 L 2 -agnostic algorithm with O ( √ L 2 log k + k log T ) bound for ℓ t ∈ [ -1 , 1] k

Here, we consider the case of ℓ t ∈ [ -1 , 1] k . It is worth noting that the negative loss cannot be handled by simply shifting the loss since it removes the sparsity from the losses ( ℓ t ) ; see [10, 27] and Appendix K for further details. We directly use the output q t as p t , that is, p t = q t . We use the hybrid regularizer consisting of the negative Shannon entropy and the log-barrier function, Φ t ( p ) = 1 η t ψ nS ( p ) + 2 ψ LB ( p ) , where ψ LB ( p ) = ∑ k i =1 log(1 /x i ) . We use learning rate η t = 1 /β t and

where ω t is defined in (6). Learning rate (7) corresponds to that in Definition 2 with h t ← H ( q 1 ) = log k , z t ← ν t , ¯ z t ← ν t , and c 2 ← 0 . We then have the following bound:

Corollary 3. If we run the above algorithm with c 1 = √ 2 , Reg T ≤ 4 √ 2 √ L 2 log k +2 k log T + k +1 / 4 , which implies that Reg T ≤ 4 √ 2 √ sT log k +2 k log T + k +1 / 4 .

The proof is given in Appendix G. Corollary 3 removes the assumption of T ≥ 7 k 2 /s 3 in Corollary 2, and it also improves the leading constant of the regret in [10]. Note that one can prove a bound of the same order, but with a worse leading constant, by setting β 1 ≥ 15 k and combining the analysis similar to that in Section 5.1.1 and the stability bound in [10]. We successfully remove the assumption of T ≥ 7 k 2 /s 3 thanks to the following lemma, which serves as one of the key elements in achieving a BOBW guarantee with a sparsity-dependent bound (The proof is given in Appendix G.):

Lemma 3 (Stability bound for negative losses) . Let ℓ t ∈ [ -1 , 1] k and ̂ y ti = ℓ ti ✶ [ A t = i ] /p ti be the inverse weighted estimator. Assume that q t ≤ δp t for some δ ≥ 1 . Then the stability term of FTRL with the hybrid regularizer Φ t = 1 η t ψ nS +2 δ ψ LB is bounded as

Remark. Wecan observe from Lemma 3 that the stability term is bounded in terms of ν t , and the most important observation is that this ν t is bounded by the inverse of the learning rate 1 / (2 η t ) = β t / 2 , i.e., ν t ≤ β t / 2 . This enables us to guarantee the stability condition (S2) in Theorem 1 without needing to mix the ˜ Θ(( kT ) -2 / 3 ) uniform exploration used in Section 5.1.1. Moreover, this will be a key property to prove a BOBW with a sparsity-dependent bound in the next section.

As a minor contribution, by directly bounding the stability component, the RHS of Lemma 3 has a smaller leading constant than the bound obtained by using the bound in [10].

5.2 Best-of-both-worlds guarantee with sparsity-dependent bound

Finally, we are ready to establish a BOBW algorithm with a sparsity-dependent bound and derive its regret bound. We use p t = T t ( q t ) = (1 -γ ) q t + γ k 1 with γ = k T ( i.e., Θ(1 /T ) uniform exploration)

and assume γ ∈ [0 , 1 / 2] , which implies 2 p t ≥ q t and ν t ≤ T. We use the hybrid regularizer Φ t = 1 η t ψ nS +4 ψ LB with learning rate η t = 1 /β t and

where ν t is defined in (7). Note that this learning depends on both the stability and penalty components. This corresponds to the SPA learning rate in Definition 2 with z t ← ν t , ¯ z t ← ν t h t +1 /h 1 , and c 2 ← 81 c 2 1 . One can see that stability assumption (S1) in Part (I) of Theorem 1 are satisfied thanks to ν t ≤ β t (see Appendix H for the proof). We then have the following bounds.

Theorem 4 (BOBW with sparsity-dependent bound) . Suppose that T ≥ 2 k . Then the above algorithm with c 1 = √ 2 log (1 + T/β 1 ) (Algorithm 2 in Appendix H) achieves

in the adversarial regime,

in the adversarial regime with a (∆ , C, T ) self-bounding constraint, and Reg T = O ( E [ ∑ k i =1 ℓ 2 ti ] log( T ) log( kT ) / ∆ min ) in the stochastic regime.

The proof is given in Appendix H. This is the first BOBW bound with the sparsity-dependent ( L 2 -dependent) bound. Note that the bounds in the stochastic regime and the adversarial regime with a self-bounding constraint can also exploit the propoerty of the underlying losses. The bound in the stochastic regime is suboptimal in two respects: its dependence on ∆ min and (log T ) 2 . Concurrently, two separate studies improve each suboptimality ([25] for ∆ min and [13] for (log T ) 2 ), but it is highly uncertain if we can prove a BOBW with a sparsity-dependent bound based on their approach, and it is an important future work to investigate this problem.

Key elements of the proof In the following, we describe some key elements of the proof of Theorem 4. We need to solve one remaining technical issue. Using Part (I) of Theorem 1, we can show that the regret is roughly bounded by E [√ ∑ T t =1 ν t h t +1 ] ≤ √ ∑ T t =1 E [ ν t h t +1 ] . However, this quantity cannot be straightforwardly bounded since h t +1 depends on ν t .

To address this issue, we analyze the behavior of arm selection probabilities when the regularizer changes. In particular, we first prove in Lemma 7 that h t +1 ≤ h t + k ( β t +1 /β t -1) h t +1 . This lemma can be proven by a novel analysis evaluating the changes of the FTRL outputs when the learning rate varies (given in Appendices H.1 and H.2) which is not considered and required when we use a time-invariant learning rate ( e.g., [10]). Using the last in-

for a, b, x > 0 implies x ≲ √ b + a (here we ignore some logarithmic factors). This combined with the self-bounding technique leads to a BOBW guarantee in the stochastic regime.

Implementation One may wonder how to compute β t +1 satisfying (8) since h t +1 = h t +1 ( β t +1 ) depends on β t +1 . In fact, this can be computed by defining F t : [ β t , β t + T ] → R as F t ( α ) = α -( β t + c 1 ν t / √ 81 c 2 1 + ν t h t +1 ( α ) + ∑ t -1 s =1 ν s h s +1 ) and setting β t +1 to be a root of F t ( α ) = 0 . Such α can be computed using the bisection method because F t is continuous (proved in Proposition 2 in Appendix H.3). The detailed discussion can be found in Appendix H.3.

6 Best-of-both-worlds with game-dependent bound for partial monitoring

This section discusses the result of a BOBW guarantee with a game-dependent bound for PM. The desired bound is obtained by direct application of the SPA learning rate and Theorem 1, which highlights the usefulness of the SPA learning rate. Due to the space constraints, the background and algorithm are given in Appendix B.

We rely on the Exploration by Optimization (EbO) by Lattimore and Szepesvári [32]. Define

where η t is a learning rate, Ψ q ( z ) = ⟨ q, exp( -z ) + z -1 ⟩ and bias q ( G ; x ) = ⟨ q, L e x -∑ k a =1 G ( a, Φ ax ) ⟩ + max c ∈ Π ( ∑ k a =1 G ( a, Φ ax ) c -L cx ) is the bias of the estimator. Note that the first and third terms in (9) correspond to the stability and transformation terms. Then in EbO we choose ( p t , G t ) ∈ P k × H by

where P ′ k ( q ) is the feasible set proposed in [46]. Let opt ′ q t ( η t ) be the optimal value of the optimization problem and V ′ t = max { 0 , opt ′ q t ( η t ) } be its truncation.

Remark. It is worth noting that V ′ t = max { 0 , opt ′ q t ( η t ) } captures game difficulty the learner is facing as discussed in [32]. As discussed above, ST ( p, G ) in (9), the objective function that determines opt ′ q t ( η t ) , correspond to the components of the regret upper bound of FTRL. Hence, the smaller ST ( p, G ) can become by optimizing p and G , the smaller the regret can become, and thus opt ′ q t ( η t ) captures the game difficulty.

Now we are ready to state our result.

Corollary 5. For any non-degenerate PM games, there exists an algorithm based on a SPA learning rate achieving Reg T = O ( E [√ ∑ T t =1 V ′ t log( k ) log(1 + T ) ] + mk 2 √ log( k ) log( T ) ) in the adversarial regime, and Reg T = O ( m 2 k 4 log( T ) log( kT ) / ∆ min + √ Cm 2 k 4 log( T ) log( kT ) / ∆ min + mk 2 √ log( k ) log( T )) in the adversarial regime with a (∆ , C, T ) self-bounding constraint.

An extended result and the proof are given in Appendices B and I, respectively. Recall that V ′ t = max { 0 , opt ′ q t ( η t ) } in the bound reflects the difficulty of the game the learner is facing, rather than the worst-case difficulty of the class of the game. The bounds in both regimes are optimal up to logarithmic factors, and further detailed comparisons are given in Table 2 and Appendix B.

7 Conclusion and future work

In this paper, we established the stability-penalty-adaptive (SPA) learning rate (Definition 2), which provides the regret upper bound that jointly depends on the stability and penalty components of FTRL (Theorem 1). This learning rate combined with the technique and analysis for bounding stability terms allows us to achieve BOBW and data-dependent bounds (sparsity- and game-dependent bounds) simultaneously in MAB and PM.

There are some remaining questions. First of all, it would be important future direction to apply the SPA learning rate to other online decision-making problems or regularizers. For example, it is as to investigate online learning with feedback graphs [3], in which the Shannon entropy regularizer (or the Tsallis entropy with the exponent larger than 1 -1 / log k ) is necessary to achieve nearly optimal regret bounds. Another interesting example is to employ the Tsallis entropy as a dominant regularizer in the SPA learning rate, for instance, to improve logarithmic dependences on the regret bounds, while in this paper we only focused on the Shannon entropy. Second, it is an open question whether we can achieve the bound of O ( √ sT log( k/s )) in the sparsity-dependent bound without knowing the sparsity level s . Finally, while we only considered PM games with local observability, investigating if the game-dependent bound with the BOBW guarantee is possible for PM with global observability is important future work.

Acknowledgments and Disclosure of Funding

TT was supported by JST, ACT-X Grant Number JPMJAX210E, Japan and JSPS, KAKENHI Grant Number JP21J21272, Japan. JH was supported by JSPS, KAKENHI Grant Number JP21K11747, Japan.

References

Jacob D Abernethy, Chansoo Lee, and Ambuj Tewari. Fighting bandits with a new kind of smoothness. In Advances in Neural Information Processing Systems , volume 28, pages 2197-2205, 2015.

Chamy Allenberg, Peter Auer, László Györfi, and György Ottucsák. Hannan consistency in online learning in case of unbounded losses under partial monitoring. In International Conference on Algorithmic Learning Theory , pages 229-243, 2006.

Noga Alon, Nicolò Cesa-Bianchi, Ofer Dekel, and Tomer Koren. Online learning with feedback graphs: Beyond bandits. In Proceedings of The 28th Conference on Learning Theory , volume 40, pages 23-35, 2015.

Jean-Yves Audibert and Sébastien Bubeck. Minimax policies for adversarial and stochastic bandits. In Conference on Learning Theory , volume 7, pages 1-122, 2009.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing , 32(1):48-77, 2002.

Gábor Bartók. A near-optimal algorithm for finite partial-monitoring games against adversarial opponents. In Proceedings of the 26th Annual Conference on Learning Theory , volume 30, pages 696-710, 2013.

Gábor Bartók, Dávid Pál, and Csaba Szepesvári. Minimax regret of finite partial-monitoring games in stochastic environments. In Proceedings of the 24th Annual Conference on Learning Theory , volume 19, pages 133-154, 2011.

Gábor Bartók, Navid Zolghadr, and Csaba Szepesvári. An adaptive algorithm for finite stochastic partial monitoring. In the 29th International Conference on Machine Learning , pages 1-20, 2012.

Sébastien Bubeck and Aleksandrs Slivkins. The best of both worlds: Stochastic and adversarial bandits. In Proceedings of the 25th Annual Conference on Learning Theory , volume 23, pages 42.1-42.23, 2012.

Sébastien Bubeck, Michael Cohen, and Yuanzhi Li. Sparsity, variance and curvature in multiarmed bandits. In Proceedings of Algorithmic Learning Theory , volume 83 of Proceedings of Machine Learning Research , pages 111-127, 2018.

Sébastien Bubeck, Yuanzhi Li, Haipeng Luo, and Chen-Yu Wei. Improved path-length regret bounds for bandits. In Conference on Learning Theory , pages 508-528, 2019.

Nicolò Cesa-Bianchi, Gábor Lugosi, and Gilles Stoltz. Regret minimization under partial monitoring. Mathematics of Operations Research , 31(3):562-580, 2006.

Christoph Dann, Chen-Yu Wei, and Julian Zimmert. A blackbox approach to best of both worlds in bandits and beyond. arXiv preprint arXiv:2302.09739 , 2023.

Liad Erez and Tomer Koren. Towards best-of-all-worlds online learning with feedback graphs. In Advances in Neural Information Processing Systems , volume 34, pages 28511-28521, 2021.

Dean Foster and Alexander Rakhlin. No internal regret via neighborhood watch. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics , volume 22, pages 382-390, 2012.

Dylan J Foster, Zhiyuan Li, Thodoris Lykouris, Karthik Sridharan, and Eva Tardos. Learning in games: Robustness of fast convergence. In Advances in Neural Information Processing Systems , volume 29, pages 4734-4742, 2016.

Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences , 55(1):119-139, 1997.

Pierre Gaillard, Gilles Stoltz, and Tim van Erven. A second-order bound with excess losses. In Proceedings of The 27th Conference on Learning Theory , volume 35, pages 176-196, 2014.

Elad Hazan and Satyen Kale. Better algorithms for benign bandits. Journal of Machine Learning Research , 12(4), 2011.

William W Hogan. Point-to-set maps in mathematical programming. SIAM review , 15(3): 591-603, 1973.

Shinji Ito. Parameter-free multi-armed bandit algorithms with hybrid data-dependent regret bounds. In Proceedings of Thirty Fourth Conference on Learning Theory , volume 134, pages 2552-2583, 2021.

Shinji Ito, Taira Tsuchiya, and Junya Honda. Nearly optimal best-of-both-worlds algorithms for online learning with feedback graphs. In Advances in Neural Information Processing Systems , volume 35, 2022.

Shinji Ito, Taira Tsuchiya, and Junya Honda. Adversarially robust multi-armed bandit algorithm with variance-dependent regret bounds. In Proceedings of Thirty Fifth Conference on Learning Theory , volume 178, pages 1421-1422, 2022.

Tiancheng Jin, Longbo Huang, and Haipeng Luo. The best of both worlds: stochastic and adversarial episodic MDPs with unknown transition. In Advances in Neural Information Processing Systems , volume 34, pages 20491-20502, 2021.

Tiancheng Jin, Junyan Liu, and Haipeng Luo. Improved best-of-both-worlds guarantees for multi-armed bandits: Ftrl with general regularizers and multiple optimal arms. arXiv preprint arXiv:2302.13534 , 2023.

Junpei Komiyama, Junya Honda, and Hiroshi Nakagawa. Regret lower bound and optimal algorithm in finite stochastic partial monitoring. In Advances in Neural Information Processing Systems , volume 28, pages 1792-1800, 2015.

Joon Kwon and Vianney Perchet. Gains and losses are fundamentally different in regret minimization: The sparse case. Journal of Machine Learning Research , 17(227):1-32, 2016.

Joon Kwon, Vianney Perchet, and Claire Vernade. Sparse stochastic bandits. In Proceedings of the 2017 Conference on Learning Theory , volume 65 of Proceedings of Machine Learning Research , pages 1269-1270, 2017.

T. L. Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics , 6(1):4-22, 1985.

Tor Lattimore and Csaba Szepesvári. Cleaning up the neighborhood: A full classification for adversarial partial monitoring. In Proceedings of the 30th International Conference on Algorithmic Learning Theory , volume 98, pages 529-556, 2019.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Tor Lattimore and Csaba Szepesvári. Exploration by optimisation in partial monitoring. In Proceedings of Thirty Third Conference on Learning Theory , volume 125, pages 2488-2515, 2020.

Chung-Wei Lee, Haipeng Luo, and Mengxiao Zhang. A closer look at small-loss bounds for bandits with graph feedback. In Proceedings of Thirty Third Conference on Learning Theory , volume 125, pages 2516-2564, 2020.

Haipeng Luo and Robert E. Schapire. Achieving all with no parameters: AdaNormalHedge. In Proceedings of The 28th Conference on Learning Theory , volume 40, pages 1286-1304, 2015.

Haipeng Luo, Chen-Yu Wei, and Kai Zheng. Efficient online portfolio with logarithmic regret. In Advances in Neural Information Processing Systems , volume 31, pages 8235-8245, 2018.

Thodoris Lykouris, Vahab Mirrokni, and Renato Paes Leme. Stochastic bandits robust to adversarial corruptions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 114-122, 2018.

Brendan McMahan. Follow-the-regularized-leader and mirror descent: Equivalence theorems and L1 regularization. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , volume 15, pages 525-533, 2011.

Francesco Orabona. A modern introduction to online learning. CoRR , abs/1912.13213, 2019. URL http://arxiv.org/abs/1912.13213 .

Antonio Piccolboni and Christian Schindelhauer. Discrete prediction games with arbitrary feedback and loss (extended abstract). In Computational Learning Theory , pages 208-223, 2001.

Roman Pogodin and Tor Lattimore. On first-order bounds, variance and gap-dependent bounds for adversarial bandits. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference , volume 115 of Proceedings of Machine Learning Research , pages 894-904, 2020.

Aldo Rustichini. Minimizing regret: The general case. Games and Economic Behavior , 29(1): 224-243, 1999.

Aadirupa Saha and Pierre Gaillard. Versatile dueling bandits: Best-of-both world analyses for learning from relative preferences. In Proceedings of the 39th International Conference on Machine Learning , volume 162, pages 19011-19026, 2022.

Yevgeny Seldin and Gábor Lugosi. An improved parametrization and analysis of the EXP3++ algorithm for stochastic and adversarial bandits. In Proceedings of the 2017 Conference on Learning Theory , volume 65, pages 1743-1759, 2017.

Yevgeny Seldin and Aleksandrs Slivkins. One practical algorithm for both stochastic and adversarial bandits. In Proceedings of the 31st International Conference on Machine Learning , volume 32 of Proceedings of Machine Learning Research , pages 1287-1295, 2014.

Taira Tsuchiya, Junya Honda, and Masashi Sugiyama. Analysis and design of Thompson sampling for stochastic partial monitoring. In Advances in Neural Information Processing Systems , volume 33, pages 8861-8871, 2020.

Taira Tsuchiya, Shinji Ito, and Junya Honda. Best-of-both-worlds algorithms for partial monitoring. In Proceedings of The 34th International Conference on Algorithmic Learning Theory , 2023.

Taira Tsuchiya, Shinji Ito, and Junya Honda. Further adaptive best-of-both-worlds algorithm for combinatorial semi-bandits. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 8117-8144, 2023.

Hastagiri P Vanchinathan, Gábor Bartók, and Andreas Krause. Efficient partial monitoring with prior information. In Advances in Neural Information Processing Systems , volume 27, pages 1691-1699, 2014.

Chen-Yu Wei and Haipeng Luo. More adaptive algorithms for adversarial bandits. In Proceedings of the 31st Conference On Learning Theory , volume 75, pages 1263-1291, 2018.

Chen-Yu Wei, Yi-Te Hong, and Chi-Jen Lu. Tracking the best expert in non-stationary stochastic environments. In Advances in Neural Information Processing Systems , volume 29, pages 39723980, 2016.

Kai Zheng, Haipeng Luo, Ilias Diakonikolas, and Liwei Wang. Equipping experts/bandits with long-term memory. In Advances in Neural Information Processing Systems , volume 32, pages 5929-5939, 2019.

Alexander Zimin and Gergely Neu. Online learning in episodic Markovian decision processes by relative entropy policy search. In Advances in Neural Information Processing Systems , volume 26, pages 1583-1591, 2013.

Julian Zimmert and Yevgeny Seldin. Tsallis-INF: An optimal algorithm for stochastic and adversarial bandits. Journal of Machine Learning Research , 22(28):1-49, 2021.

A Notation

Table 3 summarizes the symbols used in this paper.

B Detailed background and algorithm description omitted from Section 6

This section supplements the material that was briefly explained in Section 6. We also consider full information (FI) and MAB as well as non-degenerate locally observable PM (PM-local), and let M be a such underlying model.

B.1 Exploration by optimization and its extension

Exploration-by-optimization We first describe the approach of EbO by Lattimore and Szepesvári [32], which is a strong technique to bound the regret in PM with local observability. The key idea behind EbO is to minimize a part of a regret upper bound of the FTRL with the Shannon entropy. Recall that H is the set of all functions from [ k ] × Σ to R d . Then in EbO we consider the choice of ( p t , G t ) ∈ P k × H to minimize the sum of the stability and transformation terms for the worst-case outcome given as follows (also defined in (9)):

Note that the first and third terms in (11) correspond to the stability and transformation terms (divided by the learning rate η t ), respectively. We define the optimal value of the optimization problem by

Algorithm 1: BOBW algorithm with a game-dependent bound for locally observable games

opt q ( η ) = min ( p,G ) ∈ P k × H ST ( p, G ; q t , η ) and its truncation at round t by V t = max { 0 , opt q t ( η t ) } (appeared in Table 2). Note that this optimization problem is convex and can be solved numerically by using standard solvers [32].

Extending exploration-by-optimization While the vanilla EbO is a strong tool to derive a regret bound in PM-local, it only has a guarantee for the adversarial regime. Recall that in the self-bounding technique, we require a lower bound of the regret expressed in terms of q t (see Lemma 1). However, when we use the vanilla EbO, it may make a certain action selection probability p ta for some action a become extremely small even when the output of FTRL q ta is far from zero [32], which makes it impossible for us to use the self-bounding technique.

To solve this problem, the vanilla EbO was recently extended so that it is applicable to the stochastic regime (and the adversarial regime with a self-bounding constraint) for PM-local [46]. We define P ′ k ( q, M ) for a class of games M , which is the extended version of P ′ k ( q ) in (10), by

We then consider the following optimization problem, which can be seen as a slight generalization of the approach developed in [46]:

where the feasible region P k of p is replaced with P ′ k ( q, M ) . We define the optimal value of (12) by opt ′ q ( η, M ) and its truncation at round t by V ′ t ( M ) = max { 0 , opt ′ q t ( η t , M ) } . We will abbreviate M when it is clear from a context. The following proposition shows that the component of the regret in (9) can be made small enough even if the feasible region is restricted to P ′ k ( q, M ) ⊂ P k .

Proposition 1. Let M be an underlying model. If M is FI, MAB, or PM-local with η ≤ 1 / (2 mk 2 ) ,

One can immediately obtain this result by following the same lines as [32, Propositions 11 and 12] and [46, Lemma 5].

B.2 Algorithm

We use the negative Shanon entropy regularizer Φ t = 1 η t ψ nS for (1) with learning rate η t = 1 /β t with

where B = 1 / 2 for FI, B = k/ 2 for MAB, and B = 2 mk 2 for PM-local, which corresponds to the learning rate in Definition 2 with h t ← H ( q t ) , z t ← V ′ t , ¯ z t ← 0 , and c 2 ← 0 . Algorithm 1 summarizes the proposed algorithm.

B.3 Main result

Let r M be 1 if M is FI or MAB, and 2 k if M is PM-local. Then we have the following bound. Corollary 6 (Extended version of Corollary 5) . Let M be FI, MAB, or PM-local. Then the above algorithm with c 1 = √ log(1 + T ) / 2 (Algorithm 1) achieves

in the adversarial regime, and

in the adversarial regime with a (∆ , C, T ) self-bounding constraint.

The bound for the adversarial regime with a self-bounding constraint with C = 0 yields the bound in the stochastic regime, which is optimal up to logarithmic factors in FI and MAB, and has the same order as the bounds in [46, Theorem 6].

The bound for the adversarial regime has a form similar to [32] and is game-dependent in the sense that it can be bounded by the empirical difficulty V ′ t of the current game. In addition, we can also obtain the worst-case bound by replacing V ′ t with its upper bound ¯ V . This bound is optimal up to log( T ) factor in FI and log( k ) log( T ) factor in MAB, and is a factor of √ log T worse than the best known bound in [32], which can be seen as the cost for the BOBW guarantee (see also Table 2).

C Additional related work

This appendix provides additional related work that could not be included in the main body due to the numerous studies related to this paper.

Multi-armed bandits In the stochastic regime, it is known that the optimal regret is approximately expressed as Reg T = O ( k log T/ ∆ min ) [29]. In the adversarial regime (a.k.a. non-stochastic regime), it is known that the Online Mirror Descent (OMD) framework with the (negative) Tsallis entropy regularizer achieves O ( √ kT ) regret bounds [1, 4], which match the lower bound of Ω( √ kT ) [5].

Data-dependent bound In the adversarial MAB, algorithms with various data-dependent regret bounds have been developed. Typical examples of such bounds are first-order bounds dependent on the cumulative loss and second-order bounds depending on sample variances in losses. Allenberg et al. [2] provided an algorithm with a first-order regret bound of O ( √ kL ∗ log k ) for L ∗ = min a ∈ A ∑ T t =1 ⟨ ℓ t , a ⟩ . Second-order regret bounds have been shown in some studies, e.g., [10, 19, 49], In particular, Bubeck et al. [10] provided the regret bound of O ( √ Q 2 log k ) for Q 2 = ∑ T t =1 ∥ ℓ t -¯ ℓ ∥ 2 . Other examples of data-dependent bounds include path-length bounds in the form of O ( √ kV 1 log T ) for V 1 = ∑ T -1 t =1 ∥ ℓ t -ℓ t +1 ∥ 1 as well as a sparsity-dependent bound, which have been investigated [10, 11, 27, 49, 51].

Sparsity-dependent bound The study on a sparsity-dependent bound was initiated by Kwon and Perchet [27], who showed that when ℓ t ∈ [0 , 1] k , the OMD with Tsallis entropy can achieve the bound of Reg T ≤ 2 √ e √ sT log( k/s )) and prove the matching (up to logarithmic factor) lower bound of Reg T = Ω( √ sT ) when T ≥ k 3 / (4 s 2 ) . Bubeck et al. [10] also showed that OMD with a hybrid-regularizer consisting of the Shannon entropy and a log-barrier can achieve Reg T ≤ 10 √ L 2 log k +20 k log T when ℓ t ∈ [ -1 , 1] k . Zheng et al. [51] investigated the sparse MAB problem in the context of the switching regret. Although their result is not directly related to our study, they show that sparsity is useful in some cases. Note that all of these algorithms assume the knowledge of the sparsity level and do not have a BOBW guarantee. The study to exploit the sparsity was investigated also in the stochastic regime by Kwon et al. [28]. However, they define the sparsity level s by the number of arms with rewards larger than or equal to 0 ( i.e., losses smaller than or equal to 0 ), and hence the definition of sparsity is different from that in our paper.

Adaptive learning rate The stability-dependent learning rate is quite ubiquitous (see [38] and the references therein). To our knowledge, the literature on the penalty-dependent bound is quite scarce in bandits and considered in the context of BOBW algorithms [22, 46], both of which consider the Shannon entropy regularizer.

Best-of-both-worlds Since the seminal study by Bubeck and Slivkins [9], BOBW algorithms have been developed for many online decision-making problems. Although they have been investigated mainly in the context of an MAB [43, 53], other settings have also been investigated, [18, 24, 49], to name a few.

FTRL and OMD are now one of the most common approaches to achieving a BOBW guarantee owing to the usefulness of the self-bounding technique [18, 49, 53], while the first [9] and earlier work [43, 44] on BOBW do not rely on the technique. Most of the recent algorithms beyond the MABare also based on FTRL (to name a few, [24, 42, 49]).

Our BOBW algorithm with the sparsity-dependent bound can be seen as one of the studies that aim to achieve BOBW and data-dependent bound simultaneously. There is not so much existing research, and we are only aware of [21, 23, 47, 49]. They consider first-, second-order, and path-length bound, and we are the first to investigate the sparsity-dependent bound in this line of work.

Log-barrier regularizer and hybrid regularizer The log-barrier regularizer has been used in various studies (to name a few, [14, 16, 35, 40, 49]). The time-invariant log-barrier (a.k.a. constant amount of log-barrier [33]), whose properties are extensively exploited in this paper, was invented by Bubeck et al. [10] and has been used in several subsequent studies [33, 51].

Partial monitoring Starting from the work by Rustichini [41], PM has been investigated in many works in the literature [7, 12, 39]. It is known that all PM games can be classified into four classes based on their minimax regrets [7, 30]. In particular, all PM games fall into trivial, easy, hard, and hopeless games, for which its minimax regrets is 0 , Θ( √ T ) , Θ( T 2 / 3 ) , and Θ( T ) , respectively. PM has also been investigated in both the adversarial and stochastic regimes as for MAB. In the stochastic regime, there are relatively small amount of works [8, 26, 45, 48], some of which are proven to achieve an instance-dependent O (log T ) regrets for locally or globally observable games. In the adversarial regime, since the development of the FeedExp3 algorithm [12, 39], many algorithms achieving the minimax optimal regret have been developed [6, 7, 15, 31].

D Proof of Theorem 1

Proof. We first prove (5) in Part (I).

Penalty term First, we consider the penalty term ∑ T t =1 ( 1 η t +1 -1 η t ) h t +1 . By the definition of β t in (4),

where the first inequality follows from ¯ z t h 1 ≥ z t h t +1 and the second inequality follows by Lemma 8 given in Appendix J.

Stability term Next, we consider the stability term ∑ T t =1 η t z t . Using the definition of β t in (4) and defining U t = √ c 2 + ¯ z t h 1 + ∑ t -1 s =1 z s h s +1 for t ∈ { 0 } ∪ [ T ] , we bound β t from below as

where the inequality follows since ( U t ) is non-decreasing. Using the last inequality, we can bound ∑ T t =1 η t z t as

Since we have U T c 1 ( β 1 + β t ) ≥ √ c 2 +¯ z t h 1 c 1 ( β 1 + β t ) ≥ ε + z t by the assumption, a part of the last inequality is further bounded as

where the second inequality follows by Lemma 8. Combining (16) and (17) yields

Combining (14) and (18) completes the proof of (5) in Part (I).

We next prove Part (II). For the penalty term, setting h t = h 1 for all t ∈ [ T ] in (14) gives

For the stability term, since there exists a > 0 such that β t ≥ ac 1 √ h 1 √ ∑ t s =1 z s for any t ∈ [ T ] by the assumption,

Summing up the above arguments completes the proof of Part (II).

E Basic facts to bound stability terms

Here, we introduce basic facts, which are useful to bound the stability term. We have

We also have the following inequalities for ϕ nS ( x ) = x log x and ϕ LB ( x ) = log(1 /x ) , which are components of the negative Shannon entropy and log-barrier function:

It is easy to prove these facts by the standard calculus and you can find the proofs of (21) and (22) in Lemma 15 of Tsuchiya et al. [46] and Lemma 5 of Ito et al. [23], respectively.

F Proof of Corollary 2

Let Reg T ( a ) = E [ ∑ T t =1 ( ℓ tA t -ℓ ta ) ] for a ∈ [ k ] . Here we provide the complete proof of Corollary 2.

Proof of Corollary 2. Fix i ∗ ∈ [ k ] . Since p t = (1 -γ ) q t + γ k 1 , it holds that

where the last inequality follows by E [ ̂ y t | q t ] = ℓ t and the Cauchy-Schwarz inequality. Then, using the standard analysis of the FTRL described in Section 3, the first term in the last inequality is bounded as

By (19) and (21) given in Appendix E, the stability term ⟨ q t -q t +1 , ̂ y t ⟩ -D Φ t ( q t +1 , q t ) in the last inequality is bounded as

where the first inequality follows from (21), the second inequality follows by (19) with ̂ y t ≥ 0 , and the last inequality holds since ∑ k i =1 q ti ̂ y 2 ti ≤ ∑ k i =1 2 p ti ̂ y 2 ti = 2 ω t .

We will confirm that the assumptions for Part (II) of Theorem 1 are indeed satisfied. Using the definition of β t in (6), We have

where the last inequality follows since β 1 = 2 c 1 √ h 1 √ k γ and k γ ≥ ω t . Hence, stability condition (S2) in Theorem 1 is satisfied with a = 2 , and one can see that the other assumptions are trivially satisfied. Hence, by Part (II) of Theorem 1,

where in the last inequality we used h 1 ≤ log k . Now,

Summing up the above arguments and setting c 1 = 1 / 2 , we have

which completes the proof of Corollary 2.

G Proof of Corollary 3

Proof of Lemma 3. Recall that Φ t ( p ) = 1 η t ψ nS ( p ) +2 δψ LB ( p ) . Since D Φ t = 1 η t D ψ nS +2 δD ψ LB and D ψ nS ( x, y ) = ∑ k i =1 D ϕ nS ( x i , y i ) and D ψ LB ( x, y ) = ∑ k i =1 D ϕ LB ( x i , y i ) , we can bound the stability term as

where in the last inequality we used (21) and (22) with

where the first inequality follows by the definition of ̂ y t and the second inequality follows by p ti ≥ q ti /δ .

Next, we will prove that for any i ∈ [ k ] ,

Fix i ∈ [ k ] . By q ti ≤ δp ti ,

Using this and ζ ( x ) ≤ x 2 for x ∈ [ -1 2 , 1 2 ] in (20), we have for any p ti ∈ [0 , 1] that

where in the last inequality we used q ti ≤ δp ti . In particular, when p ti ≤ η t , i.e., the probability of selecting arm i is small to some extent, the last inequality can be further bounded as

On the other hand when p ti > η t , we have η t ̂ y ti ≥ -1 . Hence, by the inequality ξ ( x ) ≤ x 2 for x ≥ -1 in (19),

Hence, combining (25), (26), and (27) completes the proof of (24). Finally, by combining (23) and (24) we completes the proof of Lemma 3.

Remark. When ℓ t can be negative, the Shannon entropy regularizer alone cannot bound the stability term if the arm selection probability is small, i.e., p ti ≤ η t . Introducing a time-invariant log-barrier regularizer enables us to bound the stability term even when the arm selection probability is small. This idea was proposed by Bubeck et al. [10], who analyzed the variation of arm selection probability for the change of cumulative losses. Unlike their analysis, our proof directly analyses the stability term, enabling us to obtain the tighter regret bound. More importantly, we will utilize the property ν t ≤ O (1 /η t ) many times, which directly follows from Lemma 3, in the subsequent sections to prove the BOBW guarantee with the sparsity-dependent bound.

Now, we are ready to prove Corollary 3.

Proof of Corollary 3. Fix i ∗ ∈ [ k ] . Define p ∗ ∈ P k by

Then, using the definition of the algorithm,

where the inequality follows from the definition of p ∗ and the Cauchy-Schwarz inequality. By the standard analysis of the FTRL, described in Section 3,

∗

where in the last inequality we used the fact that p i ≥ 1 /T for all i ∈ [ k ] . For the stability term, by Lemma 3 with δ = 1 (since p t = q t ),

We will confirm that the assumptions for Part (II) of Theorem 1 are indeed satisfied. By the definition of the learning rate in (7),

Using this inequality, β t is bounded from below as

where the first inequality follows by ν t ≤ β t / 2 and the above inequality, the second inequality follows by the AM-GM inequality, and the last inequality follows from 2 √ 2 β 1 ≥ c 1 √ h 1 and √ x + √ y ≥ √ x + y for x, y ≥ 0 . Dividing the both sides by 2 , we can see stability condition (S2) in Theorem 1 is satisfied with a = 1 / 2 . One can also see that the other assumptions are trivially satisfied. Hence, by Part (II) of Theorem 1,

Using the last inequality with E [√ ∑ T t =1 ν t ] ≤ E [√ ∑ T t =1 ω t ] ≤ √ L 2 , and setting c 1 = √ 2 , we have

which completes the proof of Corollary 3.

H Proof of results in Section 5.2

Appendix H.1 provides preliminary results, which will be used to quantify the difference between q t and q t +1 in Appendix H.2 and will be used to prove the continuity of F t in Appendix H.3. Appendix H.2 proves Theorem 4 and Appendix H.3 discusses the bisection method to compute β t +1 .

H.1 Some stability results

Before proving Theorem 4, we prove several important lemmas. Consider the following three optimization problems:

with L ∈ R k + , 0 ≤ ξ ≤ min i ∈ [ k ] L i , and β, β ′ > 0 satisfying β ′ ≥ β ,

for c > 0 . Note that the outputs of FTRL with ψ ( q ) and with -H ( q ) -( c/β ) ∑ k i =1 log q i are identical since adding a constant to ψ does not change the output of the above optimization problems. In the following lemma, we investigate the relation between q and r in (28).

Lemma 4. Consider q and r in (28) . Then,

Proof. From the KKT conditions there exist µ, µ ′ ∈ R such that

which implies, by ( ∇ ψ ( q )) i = log q i -c βq i , that

for all i ∈ [ k ] . This is equivalent to

Removing L i from these equalities yields that

We will prove d µ d β > 0 . Taking derivative with respect to β of (30), we have

Multiplying ( 1 q i + c q 2 i ) -1 and summing over i ∈ [ k ] in the last equality, we have

which with the fact ∑ k i =1 d q i d β = 0 implies d µ d β > 0 . Hence, since β ≤ β ′ we have µ ≤ µ ′ .

When r i ≤ q i , it is obvious that we get r i ≤ q β/β ′ i .

When r i > q i , using (31) with the inequalities β ≤ β ′ and µ ≤ µ ′ ,

which is the desired bound.

Lemma 5. Consider p , q , and r in (28) . Then, under η := 1 /β ≤ 1 15 k , we have

Proof. By Lemma 8 of Bubeck et al. [10] we have q i ≤ 3 p i for all i ∈ [ k ] . Using this with Lemma 4, we have

H.2 Proof of Theorem 4

In this section, we will provide the proof of Theorem 4. We first see that the ratio β t /β t +1 is close to one to some extent.

Lemma 6. The learning rate β t in (8) satisfies

Proof. Recall that β t = β 1 + ∑ t -1 u =1 b u with b u = c 1 ν u U u and U t = √ c 2 + ¯ z t h 1 + ∑ t -1 s =1 z s h s +1 for t ∈ { 0 } ∪ [ T ] . It suffices to show

This indeed follows since using ν t ≤ β t / 2 we have

Finally, we are ready to prove one of the key lemmas for proving the BOBW regret bound with the sparsity-dependent bound. Recall that we have p t = ( 1 -k T ) q t + 1 T 1 and h t = 1 1 -k T H ( p t ) . Using the result in Appendix H.1, we will show that h t +1 is bounded in terms of h t .

Lemma 7. Suppose that β t is defined as (8) . Then,

Proof. Let us recall that q t and q t +1 are defined as

which corresponds to optimization problems (28) with p = q t , L = ∑ t s =1 ̂ y s , ξ = ̂ y tA t , ψ = Φ t /β t , η = 1 /β t , r = q t +1 , ψ ′ = Φ t +1 /β t +1 , and η ′ = 1 /β t +1 .

Since H is concave, by p ti = (1 -k T ) q ti + 1 T and Jensen's inequality we have

which implies h t ≥ H ( q t ) . By Lemma 5 we also have q t +1 ,i ≤ 3 q β t /β t +1 ti , which implies that

The last inequality follows since when ( 1 -k T ) 3 q β t /β t +1 ti ≤ 1 T ,

and otherwise

Using these inequalities, we have

where the first inequality follows by the concavity of H , the second inequality follows since p t +1 ,i -3 p ti ≤ ( 1 -k T ) ( q t +1 ,i -q ti ) . Defining Q t = { i ∈ [ k ] : q t +1 ,i -3 q ti ≥ 0 } , (33) is further bounded as

where the first inequality follows by p t +1 ,i ≤ 6 p β t /β t +1 t , the second follows by Lemma 6, and the last inequality follows by q t +1 ,i ≤ 3 q β t /β t +1 ti . Since for any ε > 0 , x ∈ [0 , 1] , and γ ∈ [0 , 1] , it holds that

setting γ = k/T in (35) implies that the RHS of (34) is further bounded as

where the second inequality follows by Lemma 6 and the last inequality follows by the definition of h t +1 .

Finally we are ready to prove Theorem 4.

Proof of Theorem 4. Fix i ∗ ∈ [ k ] and define p ∗ ∈ P k by

Then, using the definition of the algorithm,

where the first inequality follows since p t = (1 -γ ) q t + γ k 1 and the last inequality follows by the definition of p ∗ and γ = k T . By the standard analysis of the FTRL described in Section 3,

We first consider the penalty term. Since Φ t = 1 η t ψ nS +4 ψ LB ,

where in the last inequality we used the fact that p ∗ i ≥ 1 /T for all i ∈ [ k ] . For the stability term, by Lemma 3 with δ = 2 ,

We will confirm that the assumptions for Part (I) of Theorem 1 are indeed satisfied. By the definition of the learning rate in (8) and ν t ≤ β t / 2 ,

Hence stability condition (S1) of Theorem 1 is satisfied and one can also see that the other assumptions are trivially satisfied. Hence, by Part (I) of Theorem 1,

where in the last inequality we used ν t ≤ T and in the equality we set c 1 = √ 2 log ( 1 + T 2 β 1 ) . By summing up the above arguments and Jensen's inequality, we have

Adversarial regime We first consider the adversarial regime. Recall that E [√ ∑ T t =1 ν t ] ≤ E [√ ∑ T t =1 ω t ] ≤ √ L 2 as was done in the proof of Corollary 3. Hence (36) with h t ≤ 2 log k (since T ≥ 2 k ) yields that

Adversarial regime with a self-bounding constraint Next we consider the adversarial regime with a self-bounding constraint. We will bound a component of (36). By Lemma 7, √ E [ ∑ T t =1 ν t h t +1 ] is bounded as

where the first inequality follows by Lemma 7, the second inequality follows by ν t ≤ β t / 2 , the last inequality follows by Lemma 8. Since x ≤ √ a + bx for x > 0 implies x ≤ 2 √ a + b ,

Algorithm 3: Bisection method for computing β t +1

̸

where we used E [ ν t | h t ] ≤ E [ ∑ k i =1 p ti ( ℓ 2 ti /p ti )] = E [ ∑ i ∈ [ k ]: ℓ ti =0 p ti ( ℓ 2 ti /p ti )] ≤ s .

We consider the case of P ( a ∗ ) ≥ e , since otherwise Lemma 2 implies ∑ T t =1 H ( p t ) ≤ e log( kT ) and thus the desired bound is trivially obtained. When P ( a ∗ ) ≥ e , Lemma 2 implies that ∑ T t =1 H ( p t ) ≤ P ( a ∗ ) log( kT ) . Then from the self-bounding technique, for any λ ∈ (0 , 1] it holds that

where the first inequality follows by Lemma 1 and the second inequality follows from a √ x -bx/ 2 ≤ a 2 / (2 b ) for a, b, x ≥ 0 . Setting λ ∈ (0 , 1] to

gives the desired regret bound for the adversarial regime with a self-bounding constraint.

Stochastic regime Using E [ ν t | h t ] ≤ E [ ω t | h t ] ≤ E [ ∑ k i =1 p ti ( ℓ 2 ti /p ti )] = E [ ∑ k i =1 ℓ 2 ti ] in the second inequality of (37) and following the same arguments as the analysis for the adversarial regime with a self-bounding constraint, one can obtain the regret bound for the stochastic regime.

H.3 Discussion on bisection method for computing β t +1

This section describes the bisection method to compute β t +1 described in Section 5.2. Recall that F t : [ β t , β t + T ] → R is defined by the difference of the both sides of the update rule of ( β t ) in (8):

where h t +1 ( α ) = 1 1 -k T H ( p t +1 ( α )) , and p t +1 ( α ) is the FTRL output with the regularizer Φ t = αψ nS +4 ψ LB . Note that c 1 ν t / √ c 2 + ν t h t +1 ( α ) + ∑ t -1 s =1 ν s h s +1 ≤ c 1 ν t /c 2 ≤ T/ 9 since ν t ≤ T .

Assume that F t is continuous. Then we can see that there exists α ∈ [ β t , β t + T ] such that F t ( α ) = 0 . In fact, if p tA t = 0 then β t +1 = β t , and otherwise, we have F t ( β t ) ≤ 0 and F t ( β t + T ) > 0 . Using the intermediate value theorem with the assumption that F t is continuous, there indeed exists α ∈ [ β t , β t + T ] satisfying F t ( α ) = 0 . We can compute such α by the bisection method. In particular, we first set the range of α to [ β t , β t + T ] , and then iteratively halve it by evaluating the value of F t at the middle point. Such a bisection method (binary search) is also used in [50], although the computed target is different. The whole BOBW algorithm with the sparsity-dependent bound in Section 5.2 is given in Algorithm 2, and the concrete procedure of the bisection given in Algorithm 3.

Now, all that remains is to show that F t is continuous. To prove this, it suffices to prove that h t +1 ( α ) = 1 1 -k T H ( p t +1 ( α )) is continuous with respect to α .

Proposition 2. F t in (38) is continuous with respect to α .

Proof of Proposition 2. Take any α ∈ [ β t , β t + T ] and then consider the following optimization problem:

where Φ t +1 = αψ nS +4 ψ LB . Now using Corollary 8.1 of Hogan [20] with the fact that the solution of the above optimization problem is unique, q t +1 ( α ) is continuous with respect to α . This completes the proof since p t +1 is continuous with respect to q t +1 , 1 /T ≤ p t +1 ,i ( α ) ≤ 1 -k/T , and H ( p ) is continuous in a neighborhood of p = p t +1 ( α ) .

I Proof of Corollary 6

This section proves Corollary 6, which is the extended result of Corollary 5. Recall that B = 1 / 2 for FI, B = k/ 2 for MAB, and B = 2 mk 2 for PM-local, and r M is 1 if M is FI or MAB, and 2 k if M is PM-local, which are appeared in Appendix B. Let Reg T ( a ) = E [ ∑ T t =1 ( L A t x t -L ax t )] = E [ ∑ T t =1 ⟨ ℓ A t -ℓ a , e x t ⟩ ] for a ∈ [ k ] .

Proof. Fix i ∗ ∈ [ k ] . From Lemma 7 in Tsuchiya et al. [46], if η t > 0 , we have

We will confirm that the assumptions for Part (I) of Theorem 1 are indeed satisfied. Since

stability condition (S1) is satisfied. One can also see that the other conditions are trivially satisfied. Hence, using Part (I) of Theorem 1, we can bound the RHS of (39) as

where the second inequality follows from V ′ u / ¯ V ≤ 1 and in the equality we set c 1 = √ log(1+ T ) 2 and used √ ¯ V ≤ B .

Adversarial regime For the adversarial regime, since H ( q t ) ≤ log k , (40) immediately implies

which is the desired bound.

Adversarial regime with a self-bounding constraint Next, we consider the adversarial regime with a self-bounding constraint. We consider the case of Q ( a ∗ ) ≥ e , since otherwise Lemma 2 implies ∑ T t =1 H ( p t ) ≤ e log( kT ) and thus the desired bound is trivially obtained. When Q ( a ∗ ) ≥ e , Lemma 2 implies that ∑ T t =1 H ( q t ) ≤ Q ( a ∗ ) log( kT ) . Then from the self-bounding technique, for any λ ∈ (0 , 1]

where the first inequality follows by (40) and Lemma 1 with c ′ = r M and the second inequality follows from a √ x -bx/ 2 ≤ a 2 / (2 b ) for a, b, x ≥ 0 . Setting λ ∈ (0 , 1] to

gives the desired bound for the adversarial regime with a self-bounding constraint.

J Basic lemma

Lemma 8 ([38, Lemma 4.13]) . Let a 0 ≥ 0 , ( a t ) T t =1 be non-negative reals and f : R + → R + be a non-increasing function. Then,

We include the proof for the completeness.

Proof. Let A t = ∑ t s =0 a s . Then summing the following inequality over t completes the proof:

K Comparison of the sparsity-dependent bound and the first-order bound for negative losses

If ℓ t ∈ [0 , 1] k , the first-order bound by Wei and Luo [49] implies sparsity bounds. This, however, does not hold when ℓ t ∈ [ -1 , 1] k . In fact, let us consider the case where ℓ t is a zero vector except that only one arm's loss is -1 for some t ∈ [ T ] . Then the sparsity-dependent bound becomes O ( √ T ) . On the other hand, the first-order bound in [49] is not directly applicable, and we need to transform losses to range [0 , 1] . This implies that the first-order bound becomes O ( √ kT ) , which is worse than the sparsity-dependent bound.

Peer reviews:
comments: SUMMARY:
The paper studies the problem of designing no-regret algorithm with three types of adaptivity to the setting: sparsity, game-dependency, and Best-of-Both-Worlds. To do so, the authors propose a new adaptive learning rate, called Stability-Penalty-Adaptive learning rate for Follow the regularized leader. They provide the first $s$-agnostic regret bound in the adversarial regime, where $s$ is the sparsity of the rewards. Moreover, they design the first Best-of-Both-Worlds algorithm with a sparsity-dependent regret bound. Finally, they extend the analysis to partial monitoring.

STRENGTHS:
The paper provides new improved regret bounds for different settings. The paper provides a clear comparison with previous work. The proofs of the theoretical results are not straightforward.

WEAKNESSES:
The paper presents many different results and hence it is hard to follow the paper in some points.

The analysis of the partial monitoring setting is relegated to the last half page. This makes impossible to have an insight on the results from the main paper.

QUESTIONS:
.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

comments: SUMMARY:
This paper proposes a new scheme called SPA to configure adaptive learning rates for FTRL in MAB. Developed from a stability-&penalty-dependent generic regret bound for FTRL, SPA is designed to select learning rates to optimize the bound. SPA is then applied to three scenarios with different types of adaptivity, including sparsity, game-dependency, and BOBW. In particular, SPA eschews the need of knowing the sparsity level in advance as in previous methods, and is shown to achieve a near-optimal regret bound in both stochastic and adversarial settings. Finally, the proposed method is applied to partial monitoring, which achieves a game-dependent bound and BOBW property simultaneously.

STRENGTHS:
1. Adaptivity of FTRL is an interesting and important problem. The proposed adaptive methods may improve the applicability of FTRL in real-world scenarios.

2. This paper is technically sound with solid theoretical results.

3. This paper is well written in general and easy to follow.

WEAKNESSES:
1. The design of SPA is rather straightforward, which seems a direct consequence of existing generic regret bound for FTRL. This somehow limits the novelty of the proposed method (though the subsequent extensions to scenarios with various forms of adaptivity are interesting and non-trivial).

QUESTIONS:
See the "weaknesses".

LIMITATIONS:
Limitations have been properly discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

comments: SUMMARY:
This paper studies how to achieve BOBW regret bounds in sequential decision making problems such multi-armed bandits and partial monitoring games. To this end, this paper comes up with novel SPA learning rates for FTRL. For MAB, this paper shows FTRL with SPA could simultaneously be near-optimal for stochastic and adaptively adversarial sparse MAB, and is agnostic to the sparsity level. For PM, this paper shows it could achieve a regret bound that is both game-dependent and BOBW, and is better than previous results.

STRENGTHS:
In addition to the contribution mentioned in summary: 
1. The SPA learning rate has an interesting form that may be of interest to the community and might have broader applications.

WEAKNESSES:
1. The improvement upon the previous literature seems not big enough. The main improvement for sparse MAB seems to be removing the sparsity requirement. It is possible I underestimate the hardness of learning sparsity requirement, but since here we only need a multiplicative approximation of the sparsity, may I request the authors to explain why simple ideas such as using successive halving to guess the sparsity level wouldn't work here?

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

comments: SUMMARY:
This paper studies the Follow-the-Regularized-Leader (FTRL) framework for bandit problems. It proposes a learning rate dependent on both stability and penalty terms, which is inspired by the analysis. The proposal is helpful to achieve sparisty, game-dependent and Best of both worlds regret for bandit problems under stochastic and adversarial environments.

STRENGTHS:
- The proposed learning rate can be used to design algorithms that achieve sparsity-dependent and BOBW for Multi-arm bandit problem, and game-dependent and BOBW in partial monitoring problem.
- The proposed learning rate might be of independent interest for other online learning problems.
- The preliminaries are well-introduced to average readers.

WEAKNESSES:
- It would be better to provide comparison on regret bounds with $\Delta_{\min}$ and $L_2$ with previous results (some of them don't have these parameters) in Table 1.
- There are too many in-line equations, which might hurt presentation.

QUESTIONS:
- What would we lose if the learning rate only depends on stability **or** penalty, e.g. the sparsity or gama-denpendent property or BOBW property? It would be better to illustrate as remark for comparison.

LIMITATIONS:
No limitations are discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13

comments: SUMMARY:
This work considers the Follow-the-regularized-leader paradigm for multi-arm bandit and partial monitoring problems. It proposes a new framework for designing FTRL algorithms with a time-dependent learning rate to generate algorithms that is adaptive to the sparsity of the feedback, to the setting being stochastic/adversarial/adversarial with self-bounding constraints, and to the game difficulty of a PM problem.
Specifically, it proposes to choose the learning rate $\eta_t$ based on the penalty term (changes in the objective function due to the change of the time-dependent regularizer) and the stability term (a measure of progress) observed in previous iterations.

STRENGTHS:
The overall idea of this paper has many implications in different settings: for example, in Section 5.1, it generates algorithms with near-optimal guarantees in the setting of adversarial bandits with sparse feedback with/without restrictions on the signs of the feedback; in Section 5.2 it leads to an algorithm that is adaptive to the sparsity and to the adversarial-ness.

WEAKNESSES:
While all the notations used in this paper are defined before being used, it would be great if some context/discussion can be provided before diving into the statements and analyses. For example, what is the intuitive way to see that $opt\prime_{q_t}(\eta_t)$ captures the game difficulty? It would also be nice to explicitly describe how the learning rate is used in the FTRL updates.

QUESTIONS:
The algorithm analyzed in Section 5.2 achieves BOBW and adaptivity to sparsity in the adversarial setting. Is it possible to also have a notion of sparsity in the stochastic setting?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-29