Title: Meta-Learning Adversarial Bandit Algorithms
Venue: NeurIPS 2023
Accepted: yes
Keywords: online learning

multi-armed bandits

meta-learning

multi-task learning

bandit linear optimization
Abstract: We study online meta-learning with bandit feedback, with the goal of improving performance across multiple tasks if they are similar according to some natural similarity measure.
  As the first to target the adversarial online-within-online partial-information setting, we design meta-algorithms that combine outer learners to simultaneously tune the initialization and other hyperparameters of an inner learner for two important cases:
  multi-armed bandits (MAB) and bandit linear optimization (BLO).
  For MAB, the meta-learners initialize and set hyperparameters of the Tsallis-entropy generalization of Exp3, with the task-averaged regret improving if the entropy of the optima-in-hindsight is small.
  For BLO, we learn to initialize and tune online mirror descent (OMD) with self-concordant barrier regularizers, showing that task-averaged regret varies directly with an action space-dependent measure they induce.
  Our guarantees rely on proving that unregularized follow-the-leader combined with two levels of low-dimensional hyperparameter tuning is enough to learn a sequence of affine functions of non-Lipschitz and sometimes non-convex Bregman divergences bounding the regret of OMD.

Meta-Learning Adversarial Bandit Algorithms

Abstract

We study online meta-learning with bandit feedback, with the goal of improving performance across multiple tasks if they are similar according to some natural similarity measure. As the first to target the adversarial online-within-online partial-information setting, we design meta-algorithms that combine outer learners to simultaneously tune the initialization and other hyperparameters of an inner learner for two important cases: multi-armed bandits (MAB) and bandit linear optimization (BLO). For MAB, the meta-learners initialize and set hyperparameters of the Tsallis-entropy generalization of Exp3, with the task-averaged regret improving if the entropy of the optima-in-hindsight is small. For BLO, we learn to initialize and tune online mirror descent (OMD) with self-concordant barrier regularizers, showing that task-averaged regret varies directly with an action spacedependent measure they induce. Our guarantees rely on proving that unregularized follow-the-leader combined with two levels of low-dimensional hyperparameter tuning is enough to learn a sequence of affine functions of non-Lipschitz and sometimes non-convex Bregman divergences bounding the regret of OMD.

1 Introduction

Learning-to-learn [51] is an important area of research that studies how to improve the performance of a learning algorithm by meta-learning its parameters-e.g. initializations, step-sizes, and/or representations-across many similar tasks. The goal is to encode information from previous tasks in order to achieve better performance on future ones. Meta-learning has seen a great deal of experimental work [24, 49], practical impact [21, 29], and theoretical effort [11, 18, 22, 45, 20]. One important setting is online-within-online meta-learning [19, 31], where the learner performs a sequence of tasks, each of which has a sequence of rounds. Past work has studied the full-information setting, where the loss for every arm is revealed after each round. This assumption is not realistic in many applications, e.g. recommender systems and experimental design, where often partial or bandit feedback-only the loss of the action taken-is revealed. Such feedback can be stochastic , e.g. the losses are i.i.d. from some distribution, or adversarial , i.e. chosen by an adversary. We establish the first formal guarantees for online-within-online meta-learning with adversarial bandit feedback.

As with past full-information meta-learning results, our goal when faced with a sequence of bandit tasks will be to achieve low regret on average across them. Specifically, our task-averaged regret should (a) be no worse than that of algorithms for the single-task setting, e.g. if the tasks are not very similar, and should (b) be much better on tasks that are closely related, e.g. if the same small set of arms do well on all of them. We show that a natural way to achieve both is to initialize and tune online mirror descent (OMD), an algorithm associted with a strictly convex regularizer whose hyperparam-

eters have a significant impact on performance. Our approach works because it can learn the best hyperparameters in hindsight across tasks, which will recover OMD's worst-case optimal performance if the tasks are dissimilar but will take advantage of more optimistic settings if they are related. As generalized distances, the regularizers also induce interpretable measures of similarity between tasks.

1.1 Main contributions

We design a meta-algorithm (Algorithm 1) for learning variants of OMD-specifically those with entropic or self-concordant regularizers-that are used for adversarial bandits. This meta-algorithm combines three full-information algorithms-follow-the-leader (FTL), exponentially weighted online optimization (EWOO), and multiplicative weights (MW)-to set the initialization, step-size, and regularizer-specific parameters, respectively. It works by optimizing a sequence of functions that each upper-bound the regret of OMD on a single task (Theorem 2.1), resulting in (a) interesting notions of task-similarity because these functions depend on generalized notions of distances (Bregman divergences) and (b) adaptivity, i.e not needing to know how similar the tasks are beforehand.

Our first application is to OMD with the Tsallis regularizer [3], a relative of Exp3 [6] that is optimal for adversarial MAB. We bound the task-averaged regret by the Tsallis entropy of the estimated optimain-hindsight (Corollary 3.1), which we further extend to that of the true optima by assuming a gap between the best and second-best arms (Corollary 3.2). Both results are the first known consequences of the online learnability of Bregman divergences that are non-convex in their second arguments [31], while the latter is obtained by showing that the loss estimators of a modified algorithm identify the optimal arm w.h.p. As an example, our average m -round regret across T tasks under the gap assumption is

where d is the number of actions and H β is the Tsallis entropy [52, 3] of the distribution of the optimal actions ( β = 1 recovers the Shannon entropy). 1 This entropy is low if all tasks are usually solved by the same few arms, making it a natural task-similarity notion. For example, if only s ≪ d of the arms are ever optimal then H β = O ( s ) , so using β = 1 / log d in (1) yields an asymptotic task-averaged regret of O ( √ sm log d ) , dropping fast terms. For s = O d (1) this beats the minimax optimal rate of Θ( √ dm ) [5]. On the other hand, since H 1 / 2 = O ( √ d ) , the same bound recovers this rate in the worst-case of dissimilar tasks.

Lastly, we adapt our meta-algorithm to the adversarial BLO problem by setting the regularizer to be a self-concordant barrier function, as in Abernethy et al. [2]. Our bounds yield notions of task-similarity that depend on the constraints of the action space, e.g. over the sphere the measure is the closeness of the average of the estimated optima to the sphere's surface (Corollary 4.1). We also instantiate BLO on the bandit shortest-path problem (Corollary 4.2) [50, 30].

1.2 Related work

While we are the first to consider meta-learning under adversarial bandit feedback, many have studied meta-learning in various stochastic bandit settings [9, 34, 47, 48, 35, 13, 15, 41, 10]. The latter three study stochastic bandits under various task-generation assumptions, e.g. Azizi et al. [10] is in a batchwithin-online setting where the optimal arms are adversarial. In contrast, we make no distributional assumptions either within or without. Apart from this difference, the results of Azizi et al. [10] are the ones our MAB results are most easily compared to, which we do in detail in Section 3. Notably, they assume that only s ≪ d of the d arms are ever optimal across T tasks and show (roughly speaking) ˜ O ( √ sm ) asymptotic regret; we instead focus on an entropic notion of task-similarity that achieves the same asymptotic regret when specialized to their s ≪ d . However, avoiding their explicit assumption has certain advantages, e.g. robustness in the presence of o ( T ) outlier tasks (c.f. Section 3.3).

A setting that bears some similarity to online-within-online bandits is that of switching bandits [6], and more generally online learning with dynamic comparators [4, 27, 38, 7, 55]. In such problems, instead of using a static best arm as the comparator we use a piecewise constant sequence of arms, with a limited number of arm switches. The key difference between such work and ours is our assumption that task-boundaries are known; this makes the other setting more general. However, while e.g. Exp3.S [6] can indeed be applied to online meta-learning, guarantees derived

from switching costs cannot improve upon just running Tsallis-INF on each task [39, Table 1]. Furthermore, these approaches usually quantify difficulty by the number of switches, whereas we focus on task-similarity. While there exists stochastic-setting work that measures difficulty using a notion of average change in distribution across rounds [53], it does not lead to improved performance if this average change is Ω( T ) , as is the case in e.g. the s -sparse setting discussed above.

There has been a variety of work on full-information online-within-online meta-learning [32, 12], including tuning OMD [31, 19]. Doing so for bandit algorithms has many additional challenges, including (1) their inherent and high-variance stochasticity, (2) the use of non-Lipschitz and even unbounded regularizers, and (3) the lack of access to task-optima in order to adapt to deterministic, algorithm-independent task-similarity measures. Theoretically our analysis draws on the average regret-upper-bound analysis (ARUBA) framework [31], which observes that OMD can be tuned by targeting its upper bounds, which are affine functions of Bregman divergences, and provide online learning tools for doing so. Our core structural result shows that the distance generating functions ψ θ of these Bregman divergences can be tuned without interfering with meta-learning the initialization and step-size; tuning θ is critical for adapting to settings such as that of a small set of optimal arms in MAB. Doing so depends on several refinements of the original approach, including bounding the task-averaged-regret via the spectral norm of ∇ 2 ψ θ and expressing the loss of the meta-comparator using only ψ θ , rather than via its Bregman divergence as in prior work. Finally, applying our structural result requires setting-specific analysis, e.g. to show regularity w.r.t. θ or to obtain MAB guarantees in terms of the entropy of the true optimal arms. The latter is especially difficult, as Khodak et al. [31] define task-similarity via full information upper bounds, and involves applying tools from the bestarm-identification literature [1] to show that a constrained variant of Exp3 finds the optimal arm w.h.p.

2 Learning the regularizers of bandit algorithms

We consider the problem of meta-learning over bandit tasks t = 1 , . . . , T over some fixed set K ⊂ R d , a (possibly improper) subset of which is the action space A . On each round i = 1 , . . . , m of task t we play action x t,i ∈ A and receive feedback ℓ t,i ( x t,i ) for some function ℓ t,i : A ↦→ [ -1 , 1] . Note that all functions we consider will be linear and so we will also write ℓ t,i ( x ) = ⟨ ℓ t,i , x ⟩ . Additionally, we assume the adversary is oblivious within-task , i.e. it chooses losses ℓ t, 1 , . . . , ℓ t,m at time t . We will also denote x ( a ) to be the a -th element of the vector x ∈ R d , K ◦ to be the interior of K , ∂ K its boundary, and △ to be the simplex on d elements. Finally, note that all proofs can be found in the Appendix.

In online learning, the goal on a single task t is to play actions x t, 1 , . . . x t,m that minimize the regret ∑ m i =1 ℓ t,i ( x t,i ) -ℓ t,i ( ˚ x t ) , where ˚ x t ∈ arg min x ∈K ∑ m i =1 ℓ t,i ( x ) . Lifting this to the meta-learning setting, our goal as in past work [31, 19] will be to minimize the task-averaged regret : 1 T ∑ T t =1 ∑ m i =1 ℓ t,i ( x i,t ) -ℓ t,i ( ˚ x t ) . In particular, we want to use multi-task data to improve average performance as the number of tasks T →∞ . For example, we wish to attain a task-averaged regret bound of the form o T ( poly ( m )) + ˜ O ( V √ m ) + o ( √ m ) , where V ∈ R ≥ 0 is a measure of task-similarity that is small if the tasks are similar but still yields the worst-case single-task performanceO ( √ dm ) for MAB and O ( d √ m ) for BLO-if they are not.

2.1 Online mirror descent as a base-learner

In meta-learning we are commonly interested in learning a within-task algorithm or base-learner , a parameterized method that we run on each task t . A popular approach is to learn the initialization and other parameters of a gradient-based method such as gradient descent [24, 44, 36]. If the task optima are close, the best initialization should perform well after only a few steps on a new task. We take a similar approach applied to online mirror descent, a generalization of gradient descent to non-Euclidean geometries [14]. Given a strictly convex regularizer ψ : K ◦ ↦→ R , step-size η > 0 , and initialization x t, 1 ∈ K ◦ , OMDhas the iteration

where B ( x || y ) = ψ ( x ) -ψ ( y ) -⟨∇ ψ ( y ) , x -y ⟩ is the Bregman divergence of ψ . OMDrecovers online gradient descent when ψ ( x ) = 1 2 ∥ x ∥ 2 2 , in which case B ( x || y ) = 1 2 ∥ x -y ∥ 2 2 ; another example is exponentiated gradient, for which ψ ( p ) = ⟨ p , log p ⟩ is the negative Shannon entropy on probability vectors p ∈ △ and B is the KL-divergence [46]. An important property of B is that the sum over functions B ( x t ||· ) is minimized at the mean ¯ x of the points x 1 , . . . , x T .

Algorithm 1: Tunes OMD η,θ with regularizer ψ θ : K ↦→ R and step-size η > 0 , which when run over loss estimators ˆ ℓ t, 1 , . . . , ˆ ℓ t,m , yielding task-optima ˆ x t = arg min ∑ m ⟨ ˆ ℓ t,i , x ⟩ .

Input: compact set K ⊂ R , initialization x 1 ∈ K , ordered subset Θ k ⊂ R also used to index interval bounds η, η ∈ R k ≥ 0 and hyperparameters α ∈ R k ≥ 0 , scalar hyperparameters ρ > 0 and λ ≥ 0 , learners OMD η,θ : K ↦→ R d , projections c θ : K ↦→ K θ for θ ∈ Θ k do w 1 ( θ ) ← 1 and η 1 ( θ ) ← η ( θ )+ η ( θ ) 2 // initialize MW and EWOO for task t = 1 , . . . , T do sample θ t from Θ k w.p. ∝ exp( w t ) // sample from MW distribution ˆ x t ← OMD η t ( θ t ) ,θ t ( c θ t ( x t )) // run bandit OMD within-task x t +1 ← 1 t ∑ t s =1 ˆ x s // FTL update of initialization for θ ∈ Θ k do η t +1 ( θ ) ← ∫ η ( θ ) η ( θ ) v exp ( -α ( θ ) ∑ t s =1 U ( ρ ) s ( x s ,v,θ ) ) dv ∫ η ( θ ) η ( θ ) exp ( -α ( θ ) ∑ t s =1 U ( ρ ) s ( x s ,v,θ ) ) dv // EWOO step-size update w t +1 ( θ ) ← w t ( θ ) -λU t ( x t , η t ( θ ) , θ ) // MW update of tuning parameter

OMD on loss estimators ˆ ℓ t,i constructed via partial feedback forms an important class of bandit methods [6, 2, 3]. Their regularizers ψ are often non-Lipschitz, e.g. the negative entropy, or even unbounded, e.g. the log-barrier. Thus full-information results for tuning OMD, e.g. by Khodak et al. [31] and Denevi et al. [19], do not suffice. We do adapt the former's approach of online learning a sequence U t ( x , η, θ ) of affine functions of Bregman divergences from initializations x to known points in K . We are interested in them because the regret of OMD w.r.t. a comparator y is bounded by B ( y || x ) /η + O ( ηm ) [46, 25]. In our case the comparator is based on the estimated optimum ˆ x t ∈ arg min x ∈K ⟨ ˆ ℓ t , x ⟩ , where ˆ ℓ t = ∑ m i =1 ˆ ℓ t,i , resulting from running OMD on task t using initialization x ∈ K and hyperparameters η and θ , which we denote OMD η,θ ( x ) . Unlike full-information meta-learning, we use a parameter ε > 0 to constrain this optimum to lie in a subset K ε ⊂ K ◦ . Formally, we fix a point x 1 ∈ K ◦ to be the 'center'-e.g. x 1 = 1 d /d when K is the d -simplex △ -and define the projection c ε ( x ) = x 1 + x -x 1 1+ ε mapping from K to K ε . For example, c ε 1 -ε ( x ) = (1 -ε ) x + ε 1 d /d on the simplex. This projection allows us to handle regularizers ψ that diverge near the boundary, but also introduces ε -dependent error terms. In the BLO case it also forces us to tune ε itself, as initializing too close to the boundary leads to unbounded regret while initializing too far away does not take advantage of task-similarity. Thus the general upper bounds of interest are the following functions of the initialization x , the step-size η > 0 , and a third parameter θ that is either β or ε , depending on the setting (MAB or BLO):

Here B θ is the Bregman divergence of ψ θ while g ( θ ) ≥ 1 and f ( θ ) ≥ 0 are tunable constants. We overload θ to be either β or ε for notational simplicity, as we will not tune them simultaneously; if θ = β (for MAB) then c θ ( x ) = x 1 + x -x 1 1+ ε for fixed ε , while if θ = ε (for BLO) then B θ is the Bregman divergence of a fixed ψ . The reason to optimize this sequence of upper bounds U t is because they directly bound the task-averaged regret while being no worse than the worst-case single-task regret. Furthermore, an average over Bregman divergences is minimized at the average ˆ ¯ x = 1 T ∑ T t =1 ˆ x t , where it attains the value ˆ V 2 θ = 1 T ∑ T t =1 ψ θ ( c θ ( ˆ x t )) -ψ θ ( c θ ( ˆ ¯ x )) (c.f. Claim A.1). We will show that this quantity leads to intuitive and interpretable notions of task-similarity in all the applications we study.

2.2 A meta-algorithm for tuning bandit algorithms

To learn these functions U t ( x , η, θ ) -and thus to meta-learn OMD η,θ ( x ) -our meta-algorithm sets x to be the projection c θ of the mean of the estimated optima-i.e. follow-the-leader (FTL) over the Bregman divergences in (3)-while simultaneously setting η via EWOO and θ via discrete multiplicative weights (MW). We choose FTL, EWOO, and MW because each is well-suited to the way U t depends on x , η , and θ , respectively. First, the only effect of x on U t is via the Bregman

divergence B θ ( c θ ( ˆ x t ) || x ) , over which FTL attains logarithmic regret [31]. For η , U t is exp-concave on η > 0 so long as the first term is nonzero, but it is also non-Lipschitz; the EWOO algorithm is one of the few methods with logarithmic regret on exp-concave losses without a dependence on the Lipschitz constant [26], and we ensure the first term is nonzero by regularizing the upper bounds as follows for some ρ > 0 and D 2 θ = max x , y ∈K θ B θ ( x || y ) :

Note that this function is fully defined after obtaining ˆ x t by running OMD on task t , which allows us to use full-information MW to tune θ across the grid Θ k . Showing low regret w.r.t. any θ ∈ Θ ⊃ Θ k then just requires sufficiently large k and Lipschitzness of U t w.r.t. θ . Combining all three algorithms together thus yields the guarantee in Theorem 2.1, which is our main structural result. It implies a generic approach for obtaining meta-learning algorithms by (1) bounding the task-averaged regret by an average of functions of the form U t , (2) applying the theorem to obtain a new bound o T (1) + min θ,η ˆ V 2 θ η + ηg ( θ ) m + f ( θ ) m , and (3) bounding the estimated task-similarity ˆ V 2 θ by an interpretable quantity. Crucially, since we can choose any η > 0 , the asymptotic regret is always as good as the worst-case guarantee for running the base-learner separately on each task.

Theorem 2.1 (c.f. Thm. A.1) . Suppose x 1 = arg min x ∈K ψ θ ( x ) ∀ θ and let D , M , F , and S be maxima over θ of D θ , D θ √ g ( θ ) m , f ( θ ) , and ∥∇ 2 ψ θ ∥ 2 , respectively. For each ρ ∈ (0 , 1) we can set η , η , α , and λ s.t. the expected average of the losses U t ( c θ t ( x t ) , η t ( θ t ) , θ t ) of Algorithm 1 is at most

Here ˆ V 2 θ = 1 T ∑ T t =1 ψ θ ( c θ ( ˆ x t )) -ψ θ ( c θ ( ˆ ¯ x )) and L η bounds the Lipschitz constant w.r.t. θ at ˆ V 2 θ /η + ηg ( θ ) m + f ( θ ) m . The same bound plus ( M/ρ + Fm ) √ 1 T log 1 δ holds w.p. ≥ 1 -δ .

We keep details of the dependence on S and other constants as they are important in applying this result, but in most cases setting ρ = 1 4 √ T yields ˜ O ( T 3 4 ) regret. While a slow rate, the losses U t are non-Lipschitz and non-convex in-general, and learning them allows us to tune θ over user-specified intervals and η over all positive numbers, which will be crucial later. At the same time, this tuning is what leads to the slow rate, as without tuning ( k = 1 , L η = 0 ) the same ρ yields ˜ O ( √ T ) regret. Lastly, while we focus on learning guarantees, we note that Algorithm 1 is reasonably efficient, requiring a 2 k single-dimensional integrals per task; this is discussed in more detail in Section A.3.

3 Multi-armed bandits

We now turn to our first application: the multi-armed bandit problem, where at each round i of task t we take action a t,i ∈ [ d ] and observe loss ℓ t,i ( a t,i ) ∈ [0 , 1] . As we are sampling actions from distributions x ∈ K = △ on the k -simplex, the inner product ⟨ ℓ t,i , x t,i ⟩ is the expected loss and the optimal arm ˚ a t on task t can be encoded as a vector ˚ x t s.t. ˚ x t ( a ) = 1 a =˚ a t .

We use as a base-learner a generalization of Exp3 that uses the negative Tsallis entropy ψ β ( p ) = 1 -∑ d a =1 p β ( a ) 1 -β for some β ∈ (0 , 1] as the regularizer; this improves regret from Exp3's O ( √ dm log d ) to the optimal O ( √ dm ) [3]. Note that -ψ β is the Shannon entropy in the limit β → 1 and its Bregman divergence B β ( x ||· ) is non-convex in the second argument. As the Tsallis entropy is non-Lipschitz at the simplex boundary, which is where the estimated and true optima ˆ x t and ˚ x t lie, we will project them using c ε 1 -ε ( x ) = (1 -ε ) x + ε 1 d /d to the set K ε 1 -ε = { x ∈ △ : min a x ( a ) ≥ ε/d } . We denote the resulting vectors using the superscript ( ε ) , e.g. ˆ x ( ε ) t = c ε 1 -ε ( ˆ x t ) , and also use △ ( ε ) = K ε 1 -ε to denote the constrained simplex. For MAB we also study two base-learners: (1) implicit exploration and (2) guaranteed exploration . The former uses low-variance loss under -estimators ˆ ℓ t,i ( a ) = ℓ t,i ( a )1 a t,i = a x t,i ( a )+ γ for γ > 0 , where x t,i ( a ) is the probability of sampling a on task t round i , to enable high probability bounds [43]. On the other hand, guaranteed exploration uses unbiased loss estimators (i.e. γ = 0 ) but constrains the action space to △ ( ε ) , which we will use to adapt to a task-similarity determined by the true optima-in-hindsight.

3.1 Adapting to low estimated entropy with high probability using implicit exploration

In our first setting, the base-learner runs OMD η t ,β t ( x t, 1 ) on γ -regularized estimators with Tsallis regularizer ψ β t , step-size η t , and initialization x t, 1 ∈ △ ( ε ) . Standard OMD analysis combined with implicit exploration analysis [43] shows (44) that the task-averaged regret is bounded w.h.p. by

The summands have the desired form of U t ( x t, 1 , η t , β t ) , so by Theorem 2.1 we can bound their average by

where ˆ V 2 β = 1 T ∑ T t =1 ψ β ( ˆ x ( ε ) t ) -ψ β ( ˆ ¯ x ( ε ) ) is the average difference in Tsallis entropies between the ( ε -constrained) estimated optima ˆ x t and their empirical distribution ˆ ¯ x = 1 T ∑ T t =1 ˆ x t , while L η is the Lipschitz constant of ˆ V 2 β η + ηd β m β w.r.t. β ∈ [ β, β ] . The specific instantiation of Algorithm 1 that (7) holds for is to do the following at each time t :

η β

The final guarantee for this procedure, given in full in Theorem B.1, follows by two properties of the Tsallis entropy -ψ β : (1) its Lipschitzness w.r.t. β ∈ [0 , 1] (c.f. Lem B.1) and (2) the fact that ˆ V 2 β is bounded by the entropy ˆ H β = -ψ β ( ˆ ¯ x ) of the empirical distribution of estimated optima (c.f. Lem B.2), which yields our first notion of task-similarity: multi-armed bandit tasks are similar if the empirical distribution of their (estimated) optimal arms has low entropy .

We exemplify the implications of Theorem B.1 in Corollary 3.1, where we consider three regimes of the lower bound β on the entropy parameter: β = 1 , i.e. always using Exp3; β = 1 / 2 , which corresponds to the optimal worst-case setting [3]; and β = 1 / log d , below which the OMD regret-upper-bound always worsens (and so it does not make sense to try β < 1 / log d ).

Corollary 3.1 (c.f. Cors. B.1, B.2, and B.3) . Suppose β = 1 and we set the initialization, step-size, and entropy parameter of Tsallis OMD with implicit exploration via Algorithm 1 as in Theorem B.1.

In all three settings, as T → ∞ the regret scales directly with the entropy of the estimated optima-in-hindsight, which is small if most tasks are estimated to be solved by one of a few arms and large if all arms are used roughly equally. Corollary 3.1 demonstrates the importance of tuning β : even if tasks are dissimilar, we asymptotically recover the worst-case optimal guarantee O ( √ dm ) in cases two and three because the entropy is at most d 1 -β 1 -β . On the other hand, if a constant s ≪ d actions are always minimizers, i.e. the empirical distribution ˆ ¯ x is s -sparse, then the last bound (10)

implies that Algorithm 1 can achieve task-averaged regret o T ( md ) + O ( √ sm log d ) . At the same time, this tuning is costly, with the last two results having an extra ˜ O ( d √ m 4 √ T ) term because of it. Furthermore, the bound of β = 1 2 has a slightly better dependence on d , m , and T compared to that of β = 1 log d due to the ( d ε ) 2 -β term in the bound (7) returned for MAB by our structural result.

We can compare the s -sparse result to Azizi et al. [10], who achieve task-averaged regret ˜ O ( m/ 3 √ T + √ sm log T ) for stochastic MAB. Despite our adversarial setting and no stipulations on how tasks are related, our bounds are asymptotically comparable if the estimated and true optima are roughly equivalent (ignoring their O ( √ log T ) -factor), as we also have ˜ O ( √ sm ) average regret as T →∞ . Their rate in the number of tasks is better, but at a cost of runtime exponential in s . Apart from generality, we believe a great strength of our results is their adaptiveness; unlike Azizi et al. [10], we do not need to know how many optimal arms there are to adapt to there being few of them.

3.2 Adapting to the entropy of the true optima-in-hindsight using guaranteed exploration

While the entropy of estimated optima-in-hindsight may be useful in some cases where we wish to actually compute the task-similarity, it is otherwise generally more desirable to adapt to an intrinsic and algorithm-independent measure, e.g. the entropy of the true optima-in-hindsight. However, doing so is difficult without further assumptions, as the optima are both hard to identify and the measure itself may not be fully defined in case of ties. Thus in this section we focus on the setting where we have a nonzero performance gap ∆ > 0 between the best and second-best arms:

̸

Assumption 3.1. For some ∆ > 0 and all tasks t ∈ [ T ] , 1 m ∑ m i =1 ℓ t,i ( a ) -ℓ t,i (˚ a t ) ≥ ∆ ∀ a =˚ a t .

This assumption is common in the best-arm identification literature [28, 1], which we adapt to show that the estimated optimal arms match the true optima, and thus so do their entropies. To do so, we switch to unbiased loss estimators, i.e. γ = 0 , and control their variance by lower-bounding the probability of selecting an arm to be at least ε d ; this can alternatively be expressed as running OMD using the regularizer ψ β + I △ ( ε ) , where for any C ⊂ R d the function I C ( x ) = 0 if x ∈ C and ∞ otherwise. Guaranteed exploration allows us extend the analysis of Abbasi-Yadkori et al. [1] to show that the estimated arm is optimal w.h.p.:

Lemma 3.1 (c.f. Lem C.1) . Suppose for ε > 0 and any β ∈ (0 , 1] we run OMD on task t ∈ [ T ] with regularizer ψ β + I △ ( ε ) . If m = ˜ Ω( d ε ∆ 2 ) then ˆ x t = ˚ x t w.p. ≥ 1 -d exp( -Ω( ε ∆ 2 m/d )) .

However, the constraint that the probabilities are at least ε d does lead to εm additional error on each task, with the upper bound on the task-averaged expected regret becoming

Moreover, we will no longer set ε = o T (1) , as this would require m to be increasing in T for the bestarm identification result of Lemma C.1 to hold. Thus, unlike in the previous section, our results will contain 'fast' terms-terms in the task-averaged regret that are o ( √ m ) but not decreasing in T nor affected by the task-similarity. They will still improve upon the Ω( √ dm ) MAB lower bound if tasks are similar, but the task-averaged regret will not converge to zero as T →∞ if the tasks are identical.

Nevertheless, the tuning-dependent component of the upper bounds in (11) has the appropriate form for our structural result-in fact we can use the same meta-algorithm (8) as for implicit exploration-and so we can again apply Theorem 2.1 to get a bound on the task-averaged regret in terms of the average difference ˆ V 2 β = 1 T ∑ T t =1 ψ β ( ˆ x ( ε ) t ) -ψ β ( ˆ ¯ x ( ε ) ) of the entropies of the ε -constrained estimated task-optima ˆ x ( ε ) t and their mean ˆ ¯ x ( ε ) . The easiest way to apply Lemma C.1 to bound ˆ V 2 β in terms of H β = 1 T ∑ T t =1 ψ β ( ˚ x t ) -ψ β ( ˚ ¯ x ) is via union bound on all T tasks to show that ˆ x t = ˚ x t ∀ t w.p. ≥ 1 -dT exp( -Ω( ε ∆ 2 m/d )) ; however, setting a constant failure probability leads to m growing, albeit only logarithmically, in T . Instead, by analyzing the worst-case best-arm identification probabilities, we show in Lemma C.2 that the expectation of ˆ V 2 β is bounded by H β +3 β ( d/ε ) 1 -β -1 1 -β exp ( -3 ε ∆ 2 m 28 d ) without resorting to m = ω T (1) . Assuming m ≥ 75 d ε ∆ 2 log d ε ∆ 2 is enough (69) to bound the second term by 56 dm . Then the final result (c.f. Thm. C.1) bounds the expected task-averaged regret as follows (ignoring terms that become o T (1) after setting ρ and k ):

If the gap ∆ is known and sufficiently large, then we can set ε = Θ( d ∆ 2 m ) to obtain an asymptotic task-averaged regret that scales only with the entropy H β and a fast term that is logarithmic in m :

Corollary 3.2 (c.f. Cor. C.3) . Suppose we set the initialization, step-size, and entropy parameter of Tsallis OMD with guaranteed exploration via Algorithm 1 as specified in Theorem C.1. If [ β, β ] = [ 1 log d , 1] and m ≥ 75 d ∆ 2 log d ∆ 2 , then setting ε = ˜ Θ ( d ∆ 2 m ) , ρ = 1 3 √ d 6 √ mT , and k = ⌈ 3 √ d 2 mT ⌉ ensures that the expected task-averaged regret is at most

Knowing the gap ∆ is a strong assumption, as ideally we could set ε without it. Note that if ε = Ω( 1 m p ) for some p ∈ (0 , 1) then the condition m ≥ 75 d ε ∆ 2 log d ε ∆ 2 only fails if m ≤ poly ( 1 ∆ ) , i.e. for gap decreasing in m . We can use this together with the fact that minimizing over η and β in our bound allows us to replace them with any value, even a gap-dependent one, to derive a gapindependent setting of ε that ensures a task-similarity-adaptive bound when ∆ is not too small and falls back to the worst-case optimal guarantee otherwise. Specifically, for indicator ι ∆ = 1 m ≥ 75 d ε ∆2 log d ε ∆2 , setting

η = Θ (√ h β (∆) d β m/β ) in (12) and using β = 1 2 if the condition ι ∆ fails yields asymptotic regret at most

Thus setting ε = Θ( d/m 2 3 ) yields the desired dependence on the entropy H β and a fast term in m :

Corollary 3.3 (c.f. Cor. C.4) . In the setting of Corollary 3.2 but with m = Ω( d 3 4 ) and unknown ∆ , using ε = Θ( √ d/m 2 3 ) ensures expected task-averaged regret at most

While not logarithmic, the gap-dependent term is still o ( √ m ) , and moreover the asymptotic regret is no worse than the worst-case optimal O ( √ dm ) . Note that the latter is only needed if ∆ = o (1 / 6 √ m ) .

The main improvement in this section is in using the entropy of the true optima, which can be much smaller than that of the estimated optima if there are a few good arms but large noise. Our use of the gap assumption for this seems difficult to avoid for this notion of task-similarity. We can also compare to Corollary 3.1 (10), which did not require ∆ > 0 and had no fast terms but had a worse rate in T ; in contrast, the O ( 1 3 √ )

As before, for s ≪ d 'good' arms we obtain O ( sm log d ) asymptotic regret, assuming the gap is not too small. Finally, we can also compare to the classic shifting regret bound for Exp3.S [6], which translated to task-averaged regret is O ( √ dm log( dmT )) . This is worse than even running OMDseparately on each task, albeit under weaker assumptions (not knowing task boundaries). It also cannot take advantage of repeated optimal arms, e.g. the case of s ≪ d good arms.

T rates above match that of the closest stochastic bandit result [10]. √

3.3 Adapting to entropic task similarity implies robustness to outliers

While we considered mainly the s -sparse setting as a way of exemplifying our results and comparing to other work such as Azizi et al. [10], the fact that our approach can adapt to the Tsallis entropy min β H β of the optimal arms implies meaningful guarantees for any low-entropy distribution over the optimal arms, not just sparsely-supported ones. One way to illustrate the importance of this is through an analysis of robustness to outlier tasks. Specifically, suppose that the s -sparsity assumption-that optima ˚ a t lie in a subset of [ T ] of size s ≪ d -only holds for all but O ( T p ) of the tasks t ∈ [ T ] , where p ∈ [0 , 1) . Then the best we can do using an asymptotic bound of ˜ O ( √ sm ) -e.g. that of Azizi et al.

[10] in the stochastic case or from naively applying min β ∈ (0 , 1] H β d β m/β ≤ esm log d to any of our previous results-is to substitute s + T p instead of s , which will only improve over the single-task bound if d = ω ( T p ) , i.e. in the regime where the number of arms increases with the number of tasks.

However, our notion of task-similarity allows us to do much better, as we can show (c.f. Prop. D.1) that in the same setting H β = O ( s + d 1 -β T β (1 -p ) ) for any β ∈ [ 1 log d , 1 2 ] . Substituting this result into e.g. Corollary 3.3 yields the same asymptotic result of O ( √ sm log d ) , although the rate in T is a very slow O ( √ dm/T 1 -p 2 log d ) . This demonstrates how our entropic notion of task-similarity simultaneously yields strong results in the s -sparse setting and is meaningful in more general settings.

4 Bandit linear optimization

Our last application is bandit linear optimization, in which at task t round i we play x t,i ∈ K in some convex K ⊂ R d and observe loss ⟨ ℓ t,i , x t,i ⟩ ∈ [ -1 , 1] . We will again use a variant of mirror descent, using a self-concordant barrier for ψ and the specialized loss estimators of Abernethy et al. [2, Alg. 1]. More information on such regularizers can be found in the literature on interior point methods [42]. We pick this class of algorithms because of their optimal dependence on the number of rounds and their applicability to any convex domain K via specific barriers ψ , which will yield interesting notions of task-similarity. Our ability to handle non-smooth regularizers via the structural result (Thm. 2.1) is even more important here, as barriers are infinite at the boundaries. Indeed, we will not learn a β parameterizing the regularizer and instead focus on tuning a boundary offset ε > 0 . Here we make use of notation from Section 2, where c ε maps points in K to a subset K ε defined by the Minkowski function (c.f. Def. E.1) centered at x 1 = arg min x ∈K ψ ( x ) .

From Abernethy et al. [2] we have an upper bound on the expected task-averaged regret of their algorithm run from initializations x t, 1 ∈ K ◦ with step-sizes η t > 0 and offsets ε t > 0 :

We can show (88) that D 2 ε = max x , y ∈K ε B ( x || y ) ≤ 9 ν 3 2 K √ S 1 ε , where ν is the self-concordance constant of ψ and S 1 = ∥∇ 2 ψ ( x 1 ) ∥ 2 is the spectral norm of its Hessian at the center x 1 of K . Restricting to tuning ε ∈ [ 1 m , 1] -which is enough to obtain constant task-averaged regret above if the estimated optima ˆ x t are identical-we can now apply Algorithm 1 via the following instantiation:

1 . sample ε t via the MW distribution ∝ exp( w t ) over the discretization Θ k of [ 1 m , 1]

2 . run OMD η t ,ε t using the initialization x t, 1 = 1 t -1 ∑ s<t c ε t (ˆ x t ) = x 1 + ∑ s<t ˆ x t -x 1 (1+ ε t )( t -1) (FTL)

3 . update EWOO at each ε ∈ Θ k with loss B ( c ε ( ˆ x t ) || x t, 1 )+ ρ 2 D 2 ε η +32 ηd 2 for D 2 ε = 9 ν 3 2 K √ S 1 ε

4 . update p t +1 using multiplicative weights with expert losses B ( c ε ( ˆ x t ) || x t, 1 ) η + εm (17)

Note the similarity to the MAB case (8), with the difference being the upper bound passed to EWOO and MW. Our structural result bounds the expected task-averaged regret as follows (c.f. Thm. E.1):

For ρ = o T (1) and k = ω T (1) this becomes o T ( poly ( m )) + E min ε ∈ [ 1 m , 1] ,η> 0 ˆ V 2 ε η +32 ηd 2 m + εm , where ˆ V 2 ε = 1 T ∑ T t =1 ψ ( c ε ( ˆ x t ) -ψ ( c ε ( ˆ ¯ x t ) . Then by tuning η we get an asymptotic ( T → ∞ ) regret of 4 d ˆ V ε √ 2 m + εm for any ε ∈ [ 1 m , 1] . Our analysis removes the explicit dependence on √ ν that appears in the single-task regret [2]; as an example, ν equals the number of inequalities defining a polytope K , as in the bandit shortest-path application below.

The remaining challenge is to interpret ˆ V 2 ε , which as we did for MAB we do via specific examples, in this case concrete action domains K . Our first example is for BLO over the unit sphere K = { x ∈ R d : ∥ x ∥ 2 ≤ 1 } using the appropriate log-barrier regularizer ψ ( x ) = -log(1 -∥ x ∥ 2 2 ) :

Corollary 4.1 (c.f. Cor. E.1) . For BLO on the sphere, Algorithm 1 has expected task-averaged regret

The bound above is decreasing in E ∥ ˆ ¯ x ∥ 2 2 , the expected squared norm of the average of the estimated optima ˆ x t . We thus say that bandit linear optimization tasks over the sphere are similar if the norm of the empirical mean of their (estimated) optima is large . This makes intuitive sense: if the tasks' optima are uniformly distributed, we should expect E ∥ ˆ ¯ x ∥ 2 2 to be small, even decreasing in d . On the other hand, in the degenerate case where the estimated optima ˆ x t are the same across all tasks t ∈ [ T ] , we have E ∥ ˆ ¯ x ∥ 2 2 = 1 , so the asymptotic task-averaged regret is 1 because we can use ε = 1 m . Perhaps slightly more realistically, if it is 1 m p -away from 1 for some power p ≥ 1 2 then setting ε = 1 √ m can remove the logarithmic dependence on m . These two regimes illustrate the importance of tuning ε .

As a last application, we apply our meta-BLO result to the shortest-path problem in online optimization [50, 30]. In its bandit variant [8, 17], at each step i = 1 , . . . , m the player must choose a path p i from a fixed source u ∈ V to a fixed sink v ∈ V in a directed graph G ( V, E ) . At the same time the adversary chooses edge-weights ℓ i ∈ R | E | and the player suffers the sum ∑ e ∈ p t ℓ i ( e ) of the weights in their chosen path p t . This can be relaxed as BLO over vectors x in a set K ⊂ [0 , 1] | E | defined by a set C of O ( | E | ) linear constraints ( a , b ) ⟨ a , x ⟩ ≤ b enforcing flows from u to v ; u to v paths can be sampled from any x ∈ K in an unbiased manner [2, Proposition 1]. On a single-instance, applying the BLO method of Abernethy et al. [2] ensures O ( | E | 3 2 √ m ) regret on this problem.

In the multi-instance setting, comprising a sequence t = 1 , . . . , T of shortest path instances with m adversarial edge-weight vectors ℓ t,i each, we can attempt to achieve better performance by tuning the same method across instances. Notably, we can view this as the problem of learning predictions [33] in the algorithms with predictions paradigm from beyond-worst-case analysis [40], with the OMD initialization on each instance being effectively a prediction of its optimal path. Our meta-learner then has the following average performance across bandit shortest-path instances:

Corollary 4.2 (c.f. Cor. E.2) . For multi-task bandit online shortest path, Algorithm 1 with regularizer ψ ( x ) = -∑ a ,b ∈C log( b -⟨ a , x ⟩ ) attains the following expected average regret across instances

Here the asymptotic regret scales with the sum across all constraints a , b ∈ C of the log of the ratio between the arithmetic and geometric means across tasks of the distances b - ⟨ a , c ε ( ˆ x t ) ⟩ from the estimated optimum flow c ε ( ˆ x t ) to the constraint boundary. As it is difficult to separate the effect of the offset ε , we do not state an explicit task-similarity measure like in our previous settings. Nevertheless, since the arithmetic and geometric means are equal exactly when all entries are equal-and otherwise the former is larger-the bound does show that regret is small when the estimated optimal flows ˆ x t for each task are at similar distances from the constraints, i.e. the boundaries of the polytope. Indeed, just as on the sphere, if the estimated optima are all the same then setting ε = 1 m again yields constant averaged regret.

5 Conclusion and limitations

We develop and apply a meta-algorithm for learning to initialize and tune bandit algorithms, obtaining task-averaged regret guarantees for both multi-armed and linear bandits that depend on natural, setting-specific notions of task similarity. For MAB, we meta-learn the initialization, step-size, and entropy parameter of Tsallis-entropic OMD and show good performance if the entropy of the optimal arms is small. For BLO, we use OMD with self-concordant regularizers and meta-learn the initialization, step-size, and boundary-offset, yielding interesting domain-specific task-similarity measures. Some natural directions for future work involve overcoming some limitations of our results: can we adapt to a notion of task-similarity that depends on the true optima without assuming a gap for MAB, or at all for BLO? Alternatively, can we design meta-learning algorithms that adapt to both stochastic and adversarial bandits, i.e. a 'best-of-both-worlds' guarantee? Beyond this, one could explore other partial information settings, such as contextual bandits or bandit convex optimization.

Acknowledgments

We thank our anonymous reviewers for helpful suggestions, especially concerning the analysis of robustness to outliers. This material is based on work supported in part by the National Science Foundation under grants CCF-1910321, FAI-1939606, IIS-1901403, SCC-1952085, and SES-1919453; the Defense Advanced Research Projects Agency under cooperative agreement HR00112020003; a Simons Investigator Award; an AWS Machine Learning Research Award; an Amazon Research Award; a Bloomberg Research Grant; a Microsoft Research Faculty Fellowship; a Google Faculty Research Award; a J.P. Morgan Faculty Award; a Facebook Research Award; a Mozilla Research Grant; a Facebook PhD Fellowship; and an NDSEG Fellowship. KYL is supported by the Israel Science Foundation (grant No. 447/20) and the Technion Center for Machine Learning and Intelligent Systems (MLIS). The work of RM was partially supported by the Israel Science Foundation grant number 1693/22.

References

Yasin Abbasi-Yadkori, Peter Bartlett, Victor Gabillon, Alan Malek, and Michal Valko. Best of both worlds: Stochastic & adversarial best-arm identification. In Proceedings of the 31st Conference On Learning Theory , 2018.

Jacob Abernethy, Elad Hazan, and Alexander Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In Proceedings of the International Conference on Computational Learning Theory , 2008.

Jacob Abernethy, Chansoo Lee, and Ambuj Tewari. Fighting bandits with a new kind of smoothness. In Advances in Neural Information Processing Systems , 2015.

Oren Anava and Zohar S. Karnin. Multi-armed bandits: Competing with optimal sequences. In Advances in Neural Information Processing Systems , 2016.

Jean-Yves Audibert, Sébastien Bubeck, and Gábor Lugosi. Minimax policies for combinatorial prediction games. In Proceedings of the International Conference on Computational Learning Theory , 2011.

Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal of Computing , 32:48-77, 2002.

Peter Auer, Pratik Gajane, and Ronald Ortner. Adaptively tracking the best bandit arm with an unknown number of distribution changes. In Proceedings of the 32nd Annual Conference on Learning Theory , 2019.

Baruch Awerbuch and Robert D. Kleinberg. Adaptive routing with end-to-end feedback: Distributed learning and geometric approaches. In Proceedings of the Thirty-Sixth Annual ACM Symposium on Theory of Computing , 2004.

Mohammad Gheshlaghi Azar, Alessandro Lazaric, Emma Brunskill, et al. Sequential transfer in multi-armed bandit with finite set of models. In Advances in Neural Information Processing Systems , 2013.

MohammadJavad Azizi, Thang Duong, Yasin Abbasi-Yadkori, András György, Claire Vernade, and Mohammad Ghavamzadeh. Non-stationary bandits and meta-learning with a small set of optimal arms. arXiv, 2022.

Maria-Florina Balcan, Avrim Blum, and Santosh Vempala. Efficient representations for lifelong learning and autoencoding. In Proceedings of the 28th Annual Conference on Learning Theory , 2015.

Maria-Florina Balcan, Mikhail Khodak, Dravyansh Sharma, and Ameet Talwalkar. Learning-tolearn non-convex piecewise-Lipschitz functions. In Advances in Neural Information Processing Systems , 2021.

Soumya Basu, Branislav Kveton, Manzil Zaheer, and Csaba Szepesvári. No regrets for learning the prior in bandits. In Advances in Neural Information Processing Systems , 2021.

Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters , 31:167-175, 2003.

Leonardo Cella, Alessandro Lazaric, and Massimiliano Pontil. Meta-learning with stochastic linear bandits. In Proceedings of the 37th International Conference on Machine Learning , 2020.

Nicolò Cesa-Bianchi and Gábor Lugosi. Prediction, Learning, and Games . Cambridge University Press, 2006.

Varsha Dani, Thomas Hayes, and Sham Kakade. The price of bandit information for online optimization. In Advances in Neural Information Processing Systems , 2008.

Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Learning to learning around a common mean. In Advances in Neural Information Processing Systems , 2018.

Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Online-within-online meta-learning. In Advances in Neural Information Processing Systems , 2019.

Simon S. Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. In Proceedings of the 9th International Conference on Learning Representations , 2021.

Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. In Advances in Neural Information Processing Systems , 2017.

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradientbased model-agnostic meta-learning algorithms. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics , 2020.

Xiequan Fan, Ion Grama, and Quansheng Liu. Hoeffding's inequality for supermartingales. Stochastic Processes and their Applications , 122(10):3545-3559, 2012.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning , 2017.

Elad Hazan. Introduction to online convex optimization. In Foundations and Trends in Optimization , volume 2, pages 157-325. now Publishers Inc., 2015.

Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning , 69:169-192, 2007.

Ali Jadbabaie, Alexander Rakhlin, and Shahin Shahrampour. Online optimization: Competing with dynamic comparators. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics , 2015.

Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics , 2015.

Yihan Jiang, Jakub Koneˇ cný, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning. arXiv, 2019.

Adam Kalai and Santosh Vempala. Efficient algorithms for online decision problems. Journal of Computer and System Sciences , 71:291-307, 2005.

Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Adaptive gradient-based metalearning methods. In Advances in Neural Information Processing Systems , 2019.

Mikhail Khodak, Renbo Tu, Tian Li, Liam Li, Maria-Florina Balcan, Virginia Smith, and Ameet Talwalkar. Federated hyperparameter tuning: Challenges, baselines, and connections to weight-sharing. In Advances in Neural Information Processing Systems , 2021.

Mikhail Khodak, Maria-Florina Balcan, Ameet Talwalkar, and Sergei Vassilvitskii. Learning predictions for algorithms with predictions. In Advances in Neural Information Processing Systems , 2022.

Branislav Kveton, Martin Mladenov, Chih-Wei Hsu, Manzil Zaheer, Csaba Szepesvári, and Craig Boutilier. Meta-learning bandit policies by gradient ascent. arXiv, 2020.

Branislav Kveton, Mikhail Konobeev, Manzil Zaheer, Chih-Wei Hsu, Martin Mladenov, Craig Boutilier, and Csaba Szepesvári. Meta-Thompson sampling. In Proceedings of the 38th International Conference on Machine Learning , 2021.

Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to learning quickly for few-shot learning. arXiv, 2017.

Haipeng Luo. CSCI 699 Lecture 13. 2017. URL https://haipeng-luo.net/ courses/CSCI699/lecture13.pdf .

Haipeng Luo, Chen-Yu Wei, Alekh Agarwal, and John Langford. Efficient contextual bandits in non-stationary worlds. In Proceedings of the 31st Annual Conference on Learning Theory , 2018.

Teodor Marinov and Julian Zimmert. The Pareto frontier of model selection for general contextual bandits. In Advances in Neural Information Processing Systems , 2021.

Michael Mitzenmacher and Sergei Vassilvitskii. Algorithms with predictions. In Tim Roughgarden, editor, Beyond the Worst-Case Analysis of Algorithms . Cambridge University Press, Cambridge, UK, 2021.

Ahmadreza Moradipari, Mohammad Ghavamzadeh, Taha Rajabzadeh, Christos Thrampoulidis, and Mahnoosh Alizadeh. Multi-environment meta-learning in stochastic linear bandits. In Proceedings of the 2022 IEEE International Symposium on Information Theory , 2022.

Yurii Nesterov and Arkadii Nemirovskii. Interior-Point Polynomial Algorithms in Convex Programming . SIAM Studies in Applied and Numerical Mathematics, 1994.

Gergely Neu. Explore no more: Improved high-probability regret bounds for non-stochastic bandits. In Advances in Neural Information Processing Systems , 2015.

Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv, 2018.

Nikunj Saunshi, Yi Zhang, Mikhail Khodak, and Sanjeev Arora. A sample complexity separation between non-convex and convex meta-learning. In Proceedings of the 37th International Conference on Machine Learning , 2020.

Shai Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends in Machine Learning , 4(2):107-194, 2011.

Amr Sharaf and Hal Daumé III. Meta-learning effective exploration strategies for contextual bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , 2021.

Max Simchowitz, Christopher Tosh, Akshay Krishnamurthy, Daniel J. Hsu, Thodoris Lykouris, Miro Dudik, and Robert E. Schapire. Bayesian decision-making under misspecified priors with applications to meta-learning. In Advances in Neural Information Processing Systems , 2021.

Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems , 2017.

Eiji Takimoto and Manfred K. Warmuth. Path kernels and multiplicative updates. Journal of Machine Learning Research , 4:773-818, 2003.

Sebastian Thrun and Lorien Pratt. Learning to Learn . Springer Science & Business Media, 1998.

Constantino Tsallis. Possible generalization of Boltzmann-Gibbs statistics. Journal of Statistical Physics , 52:479-487, 1988.

Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Proceedings of the 34th Annual Conference on Learning Theory , 2021.

Takuya Yamano. Some properties of q-logarithm and q-exponential functions in Tsallis statistics. Physica A: Statistical Mechanics and its Applications , 305:486-496, 2002.

Peng Zhao, Guanghui Wang, Lijun Zhang, and Zhi-Hua Zhou. Bandit convex optimization in non-stationary environments. Journal of Machine Learning Research , 22(125):1-45, 2021.

A Structural results

A.1 Properties of the Bregman divergence

Lemma A.1. Let ψ : C ↦→ R be a strictly convex function with max x ∈C ∥∇ 2 ψ ( x ) ∥ 2 ≤ S over a convex set C ⊂ R d over size max x ∈C ∥ x ∥ 2 ≤ K , and let B ( ·||· ) be the Bregman divergence generated by ψ . Then for any points x 1 , . . . , x T ∈ C the actions y 1 = arg min x ∈C ψ ( x ) and y t = 1 t -1 ∑ s<t x s have regret

Proof. Note that

so B ( x t || y ) is 2 SK -Lipschitz w.r.t. the Euclidean norm. Applying Khodak et al. [31, Prop. B.1] yields the result (note that its assumption of strong convexity of the regularizer can be replaced with strict convexity without changing the proof or result).

Claim A.1. Let ψ : K ↦→ R be a strictly-convex function with Bregman divergence B ( ·||· ) over a convex set K ⊂ R d containing points x 1 , . . . , x T . Then their mean ¯ x = 1 T ∑ T t =1 x t satisfies

Proof.

A.2 Tuning the step-size

Lemma A.2. Let ℓ 1 , . . . , ℓ T : R > 0 ↦→ R > 0 be a sequence of functions of form ℓ t ( x ) = B 2 t x + G 2 x for adversarially chosen B t ∈ [0 , D ] and some G > 0 . Then for any ρ ≥ 0 , the actions of EWOO [26, Fig. 4] with parameter 2 ρ 2 DG run on the modified losses B 2 t + ρ 2 D 2 x + G 2 x over the domain [ ρD G , D G √ 1 + ρ 2 ] achieves regret w.r.t. any x > 0 of

Proof. By Khodak et al. [31, Prop. C.1] the modified functions are 2 ρ 2 DG -exp-concave. Then Khodak et al. [31, Cor. C.2] with B t set to B t G , D to D G , α t = G 2 , and ε = ρD G yields the result.

Lemma A.3. For ˆ x 1 , . . . , ˆ x T ∈ ∂ K consider a sequence of functions of form

where B is the Bregman divergence of a strictly convex d.g.f. ψ : K ◦ ↦→ R and where x 1 = arg min x ∈K ψ ( x ) defines the projection c ε ( x ) = x 1 + x -x 1 1+ ε for some ε > 0 . Suppose we play x t +1 ← c ε ( 1 t ∑ t s =1 ˆ x s ) and set η t using the actions of EWOO [26, Fig. 4] with parameter 2 ρ 2 DG for some ρ, D ε > 0 s.t. B ( c ε ( ˆ x t ) || x ) ≤ D 2 ε ∀ x ∈ K ε on the functions B ( c ε ( ˆ x t ) || x t )+ ρ 2 D 2 ε η + ηG 2 m over the domain [ ρD ε G √ m , D ε G √ 1+ ρ 2 m ] , with η 1 being at the midpoint of the domain. Then U t ( x t , η t ) ≤ D ε G √ m ( 1 ρ + √ 1 + ρ 2 ) ∀ t ∈ [ T ] and

for K = max x ∈K ∥ x ∥ 2 and S ε = max x ∈K ε ∥∇ 2 ψ ( x ) ∥ 2 .

Proof. The first claim follows by directly substituting the worst-case values of η into U t ( x , η ) . For the second, apply Lemma A.2 followed by Lemma A.1:

Conclude by noting that the sum of Bregman divergence to c ε ( ˆ x t ) is minimized on their convex hull, a subset of K ε .

A.3 Computational and space complexity

Algorithm 1 implicitly maintains a separate copy of FTL for each hyperparameter in the continuous space of EWOO and the grid Θ k over the domain of θ , but explicitly just needs to average the estimated task-optima ˆ x t ; this is due to the mean-as-minimizer property of Bregman divergences and the linearity of c ε . Thus the memory it uses is O ( d + k ) , where k is size of the discretization of Θ and should be viewed as sublinear in T , e.g. for MAB with implicit exploration and BLO k = O ( 4 √ d √ T ) . Computationally, at each timestep t and for each grid point we must compute two single-dimensional integrals; the integrands are sums of upper bounds that just need to be incremented once per round, leading to a total per-iteration complexity of O ( k ) (ignoring the running of OMD). Although outside the scope of this work, it may be possible to avoid integration by tuning η with MW as well, rather than EWOO, but likely at the cost of worse regret because it would not take advantage of the exp-concavity of U ( ρ ) t .

A.4 Main structural result

Theorem A.1. Consider a family of strictly convex functions ψ θ : K ◦ ↦→ R parameterized by θ lying in an interval Θ ⊂ R of radius R Θ that are all minimized at the same x 1 ∈ K ◦ , and for ˆ x 1 , . . . , ˆ x T ∈ ∂ K consider a sequence of functions of form U t ( x , η, θ ) (3) , as well as the associated regularized upper bounds U ( ρ ) t (4) . Define the maximum divergence D = max θ ∈ Θ D θ , radius K = max x ∈K ∥ x ∥ 2 , and L η the Lipschitz constant w.r.t. θ ∈ Θ of ˆ V 2 θ η + ηg ( θ ) m + f ( θ ) m . Then Algorithm 1 with Θ k ⊂ Θ the uniform discretization of Θ s.t. max θ ∈ Θ min θ ′ ∈ Θ k | θ -θ ′ | ≤ R Θ k , ρ ∈ (0 , 1) , η ( θ ) = ρD θ √ g ( θ ) m ,

Proof. In the following proof, we first consider online learning U t ( · , · , θ ) for fixed θ ∈ Θ k . To tune η , we online learn the one-dimensional losses B θ ( c θ ( ˆ x t ) || c θ ( x t )) /η + ηg ( θ ) , where c θ ( ˆ x t ) is the ( η t ( θ ) -independent) action of FTL at time t . As discussed, the corresponding regularized losses U ( ρ ) t are exp-concave, and so running EWOO yields ˜ O ( M/ρ 2 +min { ρ 2 D 2 /η, ρM } T ) regret w.r.t. the original sequence [31, Cor. C.2]. At the same time, we show that FTL has logarithmic regret on the sequence B θ ( c θ ( ˆ x t ) ||· ) that scales with the spectral norm S of ∇ 2 ψ θ (c.f. Lem. A.1), and that the average loss of the optimal comparator is ˆ V 2 θ (c.f. Claim A.1). Thus, since we only care about a fixed comparator η , dividing by ηT yields the first and last terms (5). We run a copy of these algorithms for each θ ∈ Θ k ; since their losses are bounded by ˜ O ( M/ρ + Fm ) , textbook results for MW yield O ( √ T log k ) regret w.r.t. θ ∈ Θ k , which we then extend to Θ ⊃ Θ k using L η -Lipschitzness.

Formally, we have that

where the first inequality is the regret of multiplicative weights with step-size λ [46, Cor. 2.14] and the second is by applying Lemma A.3 for each θ . We then simplify and apply the definition of ˆ V 2 θ via Claim A.1 and conclude by applying Lipschitzness w.r.t. θ :

The w.h.p. guarantee follows by Cesa-Bianchi and Lugosi [16, Lem. 4.1].

B Implicit exploration

B.1 Properties of the Tsallis entropy

Lemma B.1. For any ε ∈ (0 , 1] and x ∈ △ s.t. x ( a ) ≥ ε d ∀ a ∈ [ d ] the β -Tsallis entropy H β ( x ) = -1 -∑ d a =1 x β ( a ) 1 -β is d log d ε -Lipschitz w.r.t. β ∈ [0 , 1] .

Proof. Let log β x = x 1 -β -1 1 -β be the β -logarithm function and note that by Yamano [54, Equation 6] we have log β x -log x = (1 -β )( ∂ b log β x +log β x log x ) ≥ 0 ∀ β ∈ [0 , 1] . Then we have for β ∈ [0 , 1) that

where the fourth inequality follows by Hölder's inequality, the fifth by subadditivity of x a for a ∈ (0 , 1] , the sixth by the fact that ∂ x (log β x -log x ) = x -β -1 /x ≤ 0 ∀ β, x ∈ [0 , 1) , and the last line by substituting β = 0 since ∂ β ( log β x -log x 1 -β ) = 2( x -x β ) -(1 -β )( x β + x ) log x x β (1 -β ) 3 ≤ 0 ∀ β ∈ [0 , 1) , x ∈ (0 , 1 /d ] . For β = 1 , applying L'Hôpital's rule yields

which is bounded on [ -2 d/e , 0] .

Lemma B.2. Consider x 1 , . . . , x T ∈ △ s.t. x t ( a t ) = 1 for some a t ∈ [ d ] , and let ¯ x = 1 T ∑ T t =1 x t be their average. For any ε ∈ (0 , 1] and β ∈ (0 , 1] we have that for every t ∈ [ T ]

where recall that x ( ε ) = c ε 1 -ε ( x ) = 1 d /d +(1 -ε )( x -1 d /d ) = (1 -ε ) x + ε d 1 d .

Proof. Assume w.l.o.g. that ¯ x (1) ≤ ¯ x (2) ≤ . . . ≤ ¯ x ( d ) and a t = 1 , so that x ( ε ) t = e ( ε ) 1 . We take the derivative

By the assumption that ¯ x ( a ) is non-decreasing in a , each of the summands above become non-positive. So for ε ∈ (0 , 1] the derivative is non-positive, and for ε → 0 + it goes to -∞ . Thus the l.h.s. of the bound is monotonically non-increasing in ε for all ε ∈ [0 , 1] . The result then follows from the fact that for ε = 0 we have H β ( (1 -ε )¯ x + ε d 1 d ) -H β ( e ( ε ) 1 ) = H β (¯ x ) .

B.2 Implicit exploration bounds

Lemma B.3. Suppose we play OMD β,η with regularizer ψ β the negative Tsallis entropy and initialization x 1 ∈ △ on the sequence of linear loss functions ℓ 1 , . . . , ℓ T ∈ [0 , 1] d . Then for any x ∈ △ we have

Proof. Note that the following proof follows parts of the course notes by Luo [37], which we reproduce for completeness. The OMD update at each step t involves the following two steps: set y t +1 ∈ △ s.t. ∇ ψ β ( y t +1 ) = ∇ ψ β ( x t ) -ηℓ t and then set x t +1 = arg min x ∈△ B β ( x , y t +1 ) [25, Algorithm 14]. Note that by Hazan [25, Equation 5.3] and nonnegativity of the Bregman divergence we have

To bound the second term, note that when ψ β is the negative Tsallis entropy we have

Plugging the following result, which follows from (1 + x ) α ≤ 1+ αx + α ( α -1) x 2 ∀ x ≥ 0 , α < 0 , into the above yields the desired bound.

Theorem B.1. In Algorithm 1, let OMD η,β be online mirror descent with the Tsallis entropy regularizer ψ β over γ -offset loss estimators, Θ k is a subset of [ β, β ] ⊂ [ 1 log d , 1] , and

where ˆ x ( ε ) t = (1 -ε ) ˆ x t + ε 1 d /d . Note that U ( ρ ) t ( x , η, β ) = U t ( x , η, β ) + ρ 2 ( d 1 -β -1) η (1 -β ) . Then there exists settings of η, η, α, λ s.t. for all ε, ρ, γ ∈ (0 , 1) we have w.p. ≥ 1 -δ that

Proof. In this setting we have g ( β ) = d β /β , f ( β ) = 0 , D 2 β = d 1 -β -1 1 -β , D ≤ √ d/ 2 , M = d √ m , F = 0 , S = ( d/ε ) 2 -β , and K = 1 . We have that

where the equality follows similarly to Luo [37] since ⟨ ˆ ℓ t,i , x t,i ⟩ = ℓ t,i ( a t,i ) -γ ∑ d a =1 ˆ ℓ t,i ( a ) , the first inequality follows by Lemma B.3 and the second by Hölder's inequality and the definitions of

where the the second inequality follows by Neu [43, Lemma 1] applied to each of the last four terms and the fifth by the definition of ℓ t,i and using max β ∈ [ 1 log d , 1] η ( β ) ≤ √ d em log d . Substituting into Theorem A.1 and simplifying yields the result except with ˆ V 2 β = 1 T ∑ T t =1 ψ β ( ˆ x ( ε ) t ) -ψ β ( ˆ ¯ x ( ε ) ) in place of ˆ H β , but the former is bounded by the latter by Lemma B.2.

Corollary B.1. Let β = β = 1 . Then w.h.p. we can ensure task-averaged regret at most

so long as mT ≥ d 2 or alternatively ensure

so long as mT

Proof. Applying Theorem B.1, simplifying, and dividing by T yields task-averaged regret at most

Set γ = 1 √ dmT . Then set ε = 3 √ d 2 mT and ρ = 1 3 √ T , and use η = √ ˆ H 1 dm + 1 3 √ dmT to get the first result. Otherwise, set ε = √ d mT and ρ = 1 4 √ T , and use the better of η = √ ˆ H 1 dm + 1 4 √ dmT and η = √ log d dm to get the second.

Corollary B.2. Let β = 1 2 and β = 1 and assume mT ≥ d 5 2 . Then w.h.p. we can ensure task-averaged regret at most

Proof. Applying Theorem B.1, simplifying, and dividing by T yields task-averaged regret at most

Corollary B.3. Let β = 1 log d and β = 1 and assume mT ≥ d 3 . Then w.h.p. we can ensure task-averaged regret at most

using k = ⌈ 4 √ d √ T ⌉ .

Proof. Applying Theorem B.1, dividing by T , and simplifying yields

Note that ˆ H β and d β β are both decreasing on β < 1 log d , so β in the chosen interval is optimal over all β ∈ (0 , 1] . Set γ = 1 √ dmT , ε = d 3 4 4 √ mT , ρ = 1 4 √ T , and use η = √ β ˆ H β md β + 1 4 √ dmT to get the result.

C Guaranteed exploration

C.1 Best-arm identification

Lemma C.1. Suppose for ε > 0 we run OMD on task t ∈ [ T ] with initialization x t, 1 ∈ △ ( ε ) , regularizer ψ β t + I △ ( ε ) for some β t ∈ (0 , 1] , and unbiased loss estimators ( γ = 0 ). If Assumption 3.1 holds and m> 28 d log d 3 ε ∆ 2 then ˆ x t = ˚ x t w.p. ≥ 1 -dκ , where κ = exp ( -3 ε ∆ 2 m 28 d ) .

Proof. We extend the proof by Abbasi-Yadkori et al. [1, Appendices B and F] to arbitrary lower bounds ε/d on the probability. First, since 0 ≤ ˆ ℓ t,i ( a ) ≤ d ε ℓ t,i ( a ) we have that

and so | ˆ ℓ t,i ( a ) -ℓ t,i ( a ) | ≤ d ε . Therefore since the variance of the estimated losses is a scaled Bernoulli we have that

We can thus apply a martingale concentration inequality of Fan et al. [23, Corollary 2.1] to the martingale difference sequence (MDS) ε d ( ˆ ℓ t,i ( a ) -ℓ t,i ( a )) ∈ [ -ε d , 1] to obtain

̸

where ∆ a = 1 m | ∑ m i =1 ℓ t,i ( a ) -min a ′ = a ∑ m i =1 ℓ t,i ( a ′ ) | is the per-arm loss gap in the last step we apply ∆ a ≤ 1 . For the symmetric MDS -ε d ≤ ℓ t,i ( a ) -ˆ ℓ t,i ( a ) ≤ 1 we have

We can then conclude that

̸

Pr( ˆ x t = ˚ x t )

̸

where the second-to-last line follows by substituting the bounds (54) and (55) into the left and right terms, respectively.

Lemma C.2. Suppose on each task t ∈ [ T ] we run OMD as in Lemma C.1. Then for any β ∈ (0 , 1] we have 1 T E ∑ T t =1 ψ β ( ˆ x ( ε ) t ) -ψ β ( ˆ ¯ x ( ε ) ) ≤ -ψ β ( ˚ ¯ x ) + 3 dκβ 1 -β ( ( d ε ) 1 -β -1 ) .

̸

̸

̸

Proof. We consider the expected divergence of the best initialization under the worst-case distribution of best arm estimation, which satisfies Lemma C.1 and (56). We have by Claim A.1 and the mean-as-minimizer property of Bregman divergences that

̸

To simplify the last expression, we define ¯ p = 1 T ∑ T t =1 p t and again apply the (weighted) mean-asminimizer property, followed by Claim A.1:

By substituting into the previous inequality, we can bound the expected divergence for the worst-case p t as follows:

̸

We use the shorthand h ( x ) = ψ β ( (1 -ε ) x + ε d 1 d ) . We have

and therefore

Finally, by convexity of h we have

so we can substitute into (59) to get

Applying Lemma B.2 completes the proof.

C.2 Guaranteed exploration bounds

Lemma C.3. Suppose we play OMD β,η with initialization x 1 ∈ △ ( ε ) , regularizer ψ β + I △ ( ε ) for some β ∈ (0 , 1] , and unbiased loss estimators ( γ = 0 ) on the sequence of loss functions ℓ 1 , . . . , ℓ T ∈ [0 , 1] d . Then for any ˚ a ∈ [ d ] we have expected regret

for ˆ x the estimated optimum of the loss estimators ˆ ℓ 1 , . . . , ˆ ℓ T .

Proof.

where the second inequality follows by optimality of ˆ x for the estimated losses ˆ ℓ t , the third by Lemma B.3 constrained to △ ( ε ) , and the fourth similarly to Theorem B.1 (note both are also effectively shown in Luo [37]).

Theorem C.1. In Algorithm 1, let OMD η,β be online mirror descent with the regularizer ψ β + I △ ( ε ) over unbiased ( γ = 0 ) loss estimators, Θ k is a subset of [ β, β ] ⊂ [ 1 log d , 1] , and

where ˆ x ( ε ) t = (1 -ε ) ˆ x t + ε 1 d /d . Note that U ( ρ ) t ( x , η, β ) = U t ( x , η, β ) + ρ 2 ( d 1 -β -1) η (1 -β ) . Then under Assumption 3.1 there exists settings of η, η, α, λ s.t. for all ε, ρ ∈ (0 , 1) we have that

for L η = ( log d ε η + ηm log 2 d ) d and h β (∆) = ( H β + 56 dm ) ι ∆ + d 1 -β -1 1 -β (1 -ι ∆ ) for ι ∆ = 1 m ≥ 75 d ε ∆2 log d ε ∆2 .

Since we have the same environment-dependent quantities as in Theorem B.1, we can substitute the above bound into Theorem A.1 and then apply the Lemma C.2 bound

where the last line follows by assuming m ≥ 75 d ε ∆ 2 log d ε ∆ 2 . If this condition does not hold, then we apply the default bound of E ˆ V 2 β ≤ = 1 T ∑ T t =1 ψ β ( ˆ x t ) -ψ β ( ˆ ¯ x ) ≤ d 1 -β -1 1 -β .

Corollary C.1. Let β = β = 1 . Then for known ∆ and assuming m ≥ 75 d ∆ 2 log d ∆ 2 we can ensure expected task-averaged regret at most

where W is the Lambert W -function, while for unknown ∆ we can ensure expected task-averaged regret at most

so long as m 2 ≥ 150 d log d .

Proof. Applying Theorem C.1 and simplifying yields

Corollary C.2. Let β = 1 2 and β = 1 . Then for known ∆ and assuming m ≥ 75 d ∆ 2 log d ∆ 2 we can ensure task-averaged regret at most

using k = ⌈ 3 d 2 mT ⌉ , while for unknown ∆ we can ensure expected task-averaged regret at most

so long as m ≥ 5 d 6 .

Proof. Applying Theorem C.1 and simplifying yields

Then substitute η = √ h β (∆) d β m/β and set ρ = 3 √ 1 d √ mT and ε = 75 d ∆ 2 m W ( m 75 ) (for known ∆ ) or ε = 3 √ 150 d 2 m 2 (otherwise).

Corollary C.3. Let β = 1 log d and β = 1 . Then for known ∆ and assuming m ≥ 75 d ∆ 2 log d ∆ 2 we can ensure task-averaged regret at most

using k = ⌈ 3 d 2 mT ⌉ , while for unknown ∆ we can ensure expected task-averaged regret at most

so long as m ≥ 5 d 6 .

Proof. Applying Theorem C.1 and simplifying yields

Then substitute η = √ h β (∆) d β m/β and set ρ = 3 √ 1 d √ mT and ε = 75 d ∆ 2 m W ( m 75 ) (for known ∆ ) or ε = 3 √ 150 d 2 m 2 (otherwise).

Corollary C.4. Let β = 1 log d and β = 1 . Then for unknown ∆ and assuming m ≥ max { d 3 4 , 56 } we can ensure task-averaged regret at most

Proof. Applying Theorem C.1 and simplifying yields

D Robustness to outliers

Proposition D.1. Suppose there exists a constant p ∈ [0 , 1] and a subset S ⊂ [ T ] of size s such that ˚ a t ∈ S for all but O ( T p ) MAB tasks t ∈ [ T ] . Then if β ∈ [ 1 log d , 1 2 ] we have H β = O ( s + d 1 -β T β (1 -p ) ) .

Proof. Define the vector e S ∈ [0 , 1] d s.t. e S [ a ] = 1 a ∈ S . Then by Claim A.1 and the mean-asminimizer property of Bregman divergences we have

where the last line follows by considering δ = 1 /T 1 -p

.

E Online learning with self-concordant barrier regularizers

E.1 General results

Lemma E.1. Let K ⊂ R d be a convex set and ψ : K ◦ ↦→ R d be a self-concordant barrier. Suppose ℓ 1 , . . . , ℓ T are a sequence of loss functions satisfying |⟨ ℓ t , x ⟩| ≤ 1 ∀ x ∈ K . Then if we run OMD with step-size η > 0 as in Abernethy et al. [2, Alg. 1] on the sequence of estimators ˆ ℓ t our estimated regret w.r.t. any x ∈ K ε for ε > 0 will satisfy

Proof. The result follows from Abernethy et al. [2] by stopping the derivation on the second inequality below Equation 10.

Definition E.1. For any convex set K and any point y ∈ K , π y ( x ) = inf t ≥ 0 , y + x -y t ∈K t is the

Minkowski function with pole y .

Lemma E.2. For any x ∈ K ⊂ R d and ψ : K ◦ ↦→ R a ν -self-concordant regularizer with minimum x 1 ∈ K ◦ , the quantity ψ ( c ε ( x )) is ν √ 2 -Lipschitz w.r.t. ε ∈ [0 , 1] .

Proof. Consider any ε, ε ′ ∈ [0 , 1] s.t. ε ′ -ε ∈ (0 , 1 2 ] Note that for t = ε ′ -ε 1+ ε we have

so π c ε ′ ( x ) ( c ε ( x )) ≤ ε ′ -ε 1+ ε ≤ ε ′ -ε . Therefore by Nesterov and Nemirovskii [42, Prop. 2.3.2] we have

where for the last inequality we used -log(1 -x ) ≤ x √ 2 for x ∈ [0 , 1 2 ] . The case of ε ′ -ε ∈ (0 , 1] follows by considering ε ′′ = ε ′ + ε 2 and applying the above twice.

Theorem E.1. In Algorithm 1, let OMD η,ε be online mirror descent over loss estimators specified in Abernethy et al. [2] with a ν -self-concordant barrier regularizer ψ : K ◦ ↦→ R that satisfies ν ≥ 1 and ∥∇ 2 ψ ( x 1 ) ∥ 2 = S 1 ≥ 1 . Let Θ k be a subset of [ 1 m , 1] and

Note that U ( ρ ) t ( x , η, ε ) = U t ( x , η, ε ) + 9 ν 3 2 ρ 2 Km √ S 1 η . Then there exists settings of η, η, α, λ s.t. for all ε, ρ ∈ (0 , 1) we have expected task averaged regret at most

Proof. Let ε = 1 m . For any ε ∈ [ ε, 1] and x ∈ K we have π x 1 ( c ε ( x )) ≤ 1 1+ ε , so by Nesterov and Nemirovskii [42, Prop. 2.3.2] we have

where the first inequality follows by Nesterov and Nemirovskii [42, Prop. 2.3.2] and the definition of a self-concordant barrier [2, Def. 5]. In addition, we have g ( ε ) = 32 d 2 , f ( ε ) = ε , M = 12 d √ 2 Km/ε 4 √ ν 3 S 1 , and F = 1 . We have

where the first inequality follows by Abernethy et al. [2, Lem. 8], the second by Abernethy et al. [2, Lem. 3], the third by optimality of ˆ x t , and the fourth by Lemma E.1. Substituting into Theorem A.1 and simplifying yields the result.

E.2 Specialization to the unit sphere

Corollary E.1. Let K be the unit sphere with the self-concordant barrier ψ ( x ) = -log(1 -∥ x ∥ 2 2 ) . Then Algorithm 1 attains expected task-averaged regret bounded by

Proof. Using the fact the ν = 1 and K = S 1 = 2 , we apply Theorem E.1 and simplify to obtain

Then substitute η = ˆ V ε 4 √ 2 dm + m d 4 √ T , set ρ = 1 4 √ T , and note that

where we use the fact that ∥ ˆ x t ∥ 2 = 1 and the inequality is Jensen's.

E.3 Specialization to polytopes, specifically the bandit online shortest-path problem

Corollary E.2. Let K = { x ∈ [0 , 1] | E | : ⟨ a , x ⟩ ≤ b ∀ ( a , b ) ∈ C} be the set of flows from u to v on a graph G ( V, E ) , where C ⊂ R | E | × R is a set of O ( | E | ) linear constraints. Suppose we see T instances of the bandit online shortest path problem with m timesteps each. Then sampling from probability distributions over paths from u to v returned by running Algorithm 1 with regularizer ψ ( x ) = -∑ a ,b ∈C log( b -⟨ a , x ⟩ ) attains the following expected average regret across instances

using k = ⌈ √ T ⌉ .

Proof. Using the fact that d = | E | , ν = O ( | E | ) , K = √ | E | , and S 1 ≤ ∑ a ,b ∈C ∥ aa T ∥ 2 ( ⟨ a , 1 | E | / | E |⟩-b ) 2 = O ( | E | 3 ) , we apply Theorem E.1 and simplify to obtain

Then substitute η = ˆ V ε 4 √ 2 dm + | E | 2 √ m 4 √ T , set ρ = 4 √ | E | T 6 √ m , and note that

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers learning several adversarial tasks with different loss functions simultaneously, hoping to attain better task-averaged regret if the tasks are \"similar\" enough (e.g., the optimal actions of all tasks concentrate on a small subset). A general meta-learning framework is derived by deploying three different algorithms as meta-learners to optimize the hyper-parameters of the base learners (OMD). It then applies to MABs and BLOs, providing various high-probability or expected regret bounds.\n\nSTRENGTHS:\n1. The studied problem is well-motivated. According to the authors, while the stochastic variant of this meta-learning problem is well-studied, the adversarial version tackled by this paper has never been solved.\n2. The algorithmic idea is well-illustrated and thus the framework is easy to understand.\n3. The framework is applied to various base-learners to prove its effectiveness, including MABs with implicit exploration or guaranteed exploration and BLOs with self-concordant barriers.\n\nWEAKNESSES:\n1. The design of the meta-learner does not look pretty exciting. Given the expression of $U_t$ in Eq. (3), it appears unsurprising that the adopted optimizers (FTL, EWOO, and MW) can be applied to their respective objectives.  \nIn other words, it seems that the technical contribution of this paper is not solid enough -- the analyses of meta-learners and base-learners are both not novel. However, I do feel the overall result is interesting.\n2. I am unsure whether the task similarity measure can generalize over the \"support\" of the optimizers.  \nConsider a special case where most of the tasks share the same optimizer but there are a few (say $\\sqrt T$) outliers, would the bound scale with $\\sqrt[4]{T}$ or $\\mathcal O(1)$? In this case, if a meta-learner successfully rules out those outliers, the bound should scale by $\\mathcal O(1)$; but using the sparsity $s$ would only give $\\sqrt[4]{T}$.  \nIntuitively, as $H_\\beta$ is taken to the average of all $\\hat x$'s, such a distribution will be very different from a uniform distribution (i.e., there are $\\sqrt T$ optimizers each associated with $\\sqrt T$ tasks); though I'm not sure whether the current framework can capture this.\n3. (minor) The bounds in the main text contain so many terms that it is hard to interpolate. Informal expressions (like Eq. (13) or Eq. (19)) can be shortened using $o_T(1)$ to highlight the main terms.\n\nQUESTIONS:\nSee Weakness 1 & 2.\n\nLIMITATIONS:\nClearly stated", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focused on online meta-learning with bandit feedback, and developed and applied a meta-algorithm for learning to initialize and tune bandit algorithms, obtaining task-average regret guarantees for both MAB and linear bandits.  Specifically, a meta-algorithm was developed for learning the variants of OMD, which was further applied to OMD with the Tsallis regularizer. Furthermore, the meta-algorithm was adapted to the adversarial BLO problem via setting the regularizer to be a self-concordant barrier function.\n\nSTRENGTHS:\n- This paper is the first to consider meta-learning under adversarial bandit feedback. \n- A meta-algorithm was designed for learning the variants of OMD, which can simultaneously tune the initialization and other hyperparameters.\n- Strong theoretical performance guarantees were presented for the proposed algorithm.  \n- The paper is well-written and easy to follow though it is heavy in theory.\n\nWEAKNESSES:\n- This paper mainly focused on the pure adversarial settings.  As the authors mentioned, there are extensive works studied the stochastic settings, and given the popularity of \"best-of-both-worlds\" settings in the community, can you envision what is the major challenges to generalize the solutions to the best-of-both-worlds settings, from both the algorithmic and performance analysis perspectives? \n- As the paper considered the applications of the proposed algorithms, it may be interesting to do some case studies with real-world applications, rather than only on theoretical applications.\n\nQUESTIONS:\n- This paper mainly focused on the pure adversarial settings.  As the authors mentioned, there are extensive works studied the stochastic settings, and given the popularity of \"best-of-both-worlds\" settings in the community, can you envision what is the major challenges to generalize the solutions to the best-of-both-worlds settings, from both the algorithmic and performance analysis perspectives?\n\nLIMITATIONS:\nn/a societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors consider an online meta-learning problem with the adversarial online-with-online partial information setting,\nwhere a learner selects parameters across T tasks with m rounds and can utilize similarity between tasks to achieve a low regret.\nFirst, the authors propose a meta algorithm which optimizes hyperparameters of the inner loop algorithm (OMD).\nAs an instance of this algorithm, the authors consider OMD for MAB and BLO (bandit linear optimization).\nThen, they prove several upper bounds of the average regret across $T$ tasks using a similarity of optimal parameters across tasks.\n\nSTRENGTHS:\n1. While existing papers considered online-with-online meta-learning in stochastic or full-information setting, this paper considers the adversarial bandit feedback setting, which is practically important.\n2. Although the proposed algorithm is based on one in the full information setting, there is novelty due to the bandit feed back (e.g., estimation of true $H_\\beta$).\n3. The authors provide regret upper bounds under several settings and results seems valid.\n\nWEAKNESSES:\n1. There is no discussion on lower bounds. Therefore, it is not clear whether the proposed method is optimal.\n\nQUESTIONS:\n1. In the related work section, the authors compare their algorithm to algorithm for dynamic regret optimization, but they discuss only the number of switches. There are algorithms whose regret bounds involve total variance of the environment (e.g. $\\Delta$ in Wei, Chen-Yu, and Haipeng Luo. \"Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach.\" Conference on learning theory, 2021.). Are these algorithms related to the problem setting in Sec 3.2? Is it possible to provide a comparison?\n2. In the stochastic setting, is there a known lower bound?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper proposes an online mirror descent approach for adversarial bandit feedback-based online meta learning, utilizing FTL for initialization, EWOO for step-size, and MW for regularizer-specific parameters. The proposed method is applied to two widely adopted applications: multi-armed bandits (MAB) with Tsallis regularizer and bandit linear optimization (BLO) with self-concordant barrier function regularizer. The authors provide theoretical analysis on asymptotic task-averaged regret for MAB and BLO.

STRENGTHS:
The paper is well-written and seems to cover an important topic in the community. I check some proofs, and they seems correct.

WEAKNESSES:
While the paper is theoretical in nature, it would be beneficial to include some empirical results to further validate the proposed approach.

QUESTIONS:
The reviewer is not very familiar with this topic and would like to see the comments and feedback from other reviewers before asking further questions.

LIMITATIONS:
It would be nice to provide empirical results ( it is not mandatory and the paper's contribution is still valuable without them).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses online meta-learning with bandit feedback to enhance performance across multiple tasks that share a natural similarity measure. The study focuses on the adversarial online-within-online partial-information setting and proposes meta-algorithms that combine outer learners to optimize the initialization and other hyperparameters of an inner learner for two cases: multi-armed bandits (MAB) and bandit linear optimization (BLO). For MAB, the meta-learners use the Tsallis-entropy generalization of Exp3, and the task-averaged regret improves when the entropy of the optima-in-hindsight is small. For BLO, the approach involves learning to initialize and tune online mirror descent (OMD) with self-concordant barrier regularizers, where the task-averaged regret is related to an action space-dependent measure induced by these regularizers. The guarantees provided in the study are based on demonstrating that unregularized follow-the-leader combined with two levels of low-dimensional hyperparameter tuning can effectively learn a sequence of affine functions of non-Lipschitz and sometimes non-convex Bregman divergences, which bound the regret of OMD.\n\nSTRENGTHS:\nThis paper designs a meta-algorithm combining FTL, EWOO, MW to set the initialization, step-size, and the regularizer. Then, it has to direct applications for MAB and BLO. Then, the authors provide rigorous average regret analysis with Tsallis entropy, which are the first results for online learning with Bregman divergences.\n\nWEAKNESSES:\nIt will be better if authors can provide some real-world applications that the designed algorithm can adapt to.\n\nQUESTIONS:\n(1) For the MAB algorithms, there may be only one served task in each round, i.e., observe the loss. In this case, how to run algorithm 1 and do we need to update the parameters of all tasks?\n\nLIMITATIONS:\n.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a meta-learning algorithmic framework for adversarial bandits, designed to fine-tune the initialization and hyperparameters of the internal learner, thereby enhancing performance across multiple tasks. The efficacy of this approach is validated via a theoretical analysis on multi-armed bandits and linear bandits. The authors also propose a task-similarity measure with significant implications concerning entropy and proximity to the task boundary.\n\nSTRENGTHS:\n- The paper addresses meta-learning under adversarial bandit feedback, which is an interesting problem by itself, and the meta-learning approach poses a solid contribution to the bandit community, with a new notion of task-similarity based on entropy.\n- The concrete steps towards building the algorithms is clear and statements in the paper seems solid.\n\nWEAKNESSES:\n- The paper does not have any numerical validations of the proposed algorithm. This puts doubts on the practicality of the proposed framework and methods. Synthetic as well as real-world examples should be tested upon, and comparisons against meta learning stochastic bandits should be discussed in the numerical experiments as well.\n- The computational complexity is not adequately addressed within the main body of the paper. Specifically, the computational overhead associated with the Multiplicative Weights (MW) method could be prohibitively high with a large grid and action space.\n- The interpretation of task-similarity lacks a sufficient explanation of its underlying motivation. Even in application cases where the authors justify the task-similarity measure via average entropy or proximity to the decision boundary, it remains unclear why this measure was chosen and how it compares to other task-similarity measures in stochastic bandit scenarios (and/or those with contamination or parameter noise). As it stands, the presentation in the paper makes it seem more like an artifact of the regret proof.\n\nQUESTIONS:\n- Could the authors elaborate on the connections to meta-learning in adversarial Reinforcement Learning (RL)? While meta-learning for adversarial bandits appears to be a novel concept, meta-learning in adversarial RL, a generalization of bandits, has been explored in the literature, as evidenced by [1] for example.\n- Could the authors provide insights on how the algorithm's implementation could be optimized for efficiency?\n- Could the authors demonstrate the effectiveness of the model and the algorithm in a well-motivated real-world setting through numerical experiments?\n- Could the authors offer a more intuitive explanation of the task-similarity measure and how it compares to existing methods, beyond comments about being 'distributional assumption-free'?\n\n[1] Lin, Zichuan, Garrett Thomas, Guangwen Yang, and Tengyu Ma. \"Model-based adversarial meta-reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020): 10161-10173.\n\nLIMITATIONS:\nThe authors have acknowledged the limitations of their work, including the absence of a gap-free task similarity criterion, the assumption of known task boundaries, and the lack of a 'best-of-both-worlds' guarantee.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}