Title: On the Minimax Regret for Online Learning with Feedback Graphs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Online learning

Feedback graphs

Multiarmed bandits
Abstract: In this work, we improve on the upper and lower bounds for the regret of online learning with strongly observable undirected feedback graphs. The best known upper bound for this problem is $\mathcal{O}\bigl(\sqrt{\alpha T\ln K}\bigr)$, where $K$ is the number of actions, $\alpha$ is the independence number of the graph, and $T$ is the time horizon. The $\sqrt{\ln K}$ factor is known to be necessary when $\alpha = 1$ (the experts case). On the other hand, when $\alpha = K$ (the bandits case), the minimax rate is known to be $\Theta\bigl(\sqrt{KT}\bigr)$, and a lower bound $\Omega\bigl(\sqrt{\alpha T}\bigr)$ is known to hold for any $\alpha$. Our improved upper bound $\mathcal{O}\bigl(\sqrt{\alpha T(1+\ln(K/\alpha))}\bigr)$ holds for any $\alpha$ and matches the lower bounds for bandits and experts, while interpolating intermediate cases. To prove this result, we use FTRL with $q$-Tsallis entropy for a carefully chosen value of $q \in [1/2, 1)$ that varies with $\alpha$. The analysis of this algorithm requires a new bound on the variance term in the regret. We also show how to extend our techniques to time-varying graphs, without requiring prior knowledge of their independence numbers. Our upper bound is complemented by an improved $\Omega\bigl(\sqrt{\alpha T(\ln K)/(\ln\alpha)}\bigr)$ lower bound for all $\alpha > 1$, whose analysis relies on a novel reduction to multitask learning. This shows that a logarithmic factor is necessary as soon as $\alpha < K$.

On the Minimax Regret for Online Learning with Feedback Graphs

Abstract

In this work, we improve on the upper and lower bounds for the regret of online learning with strongly observable undirected feedback graphs. The best known upper bound for this problem is O ( √ αT ln K ) , where K is the number of actions, α is the independence number of the graph, and T is the time horizon. The √ ln K factor is known to be necessary when α = 1 (the experts case). On the other hand, when α = K (the bandits case), the minimax rate is known to be Θ ( √ KT ) , and a lower bound Ω ( √ αT ) is known to hold for any α . Our improved upper bound O (√ αT (1 + ln( K/α )) ) holds for any α and matches the lower bounds for bandits and experts, while interpolating intermediate cases. To prove this result, we use FTRL with q -Tsallis entropy for a carefully chosen value of q ∈ [1 / 2 , 1) that varies with α . The analysis of this algorithm requires a new bound on the variance term in the regret. We also show how to extend our techniques to timevarying graphs, without requiring prior knowledge of their independence numbers. Our upper bound is complemented by an improved Ω (√ αT (ln K ) / (ln α ) ) lower bound for all α > 1 , whose analysis relies on a novel reduction to multitask learning. This shows that a logarithmic factor is necessary as soon as α < K .

1 Introduction

Feedback graphs [29] provide an elegant interpolation between two popular online learning models: multiarmed bandits and prediction with expert advice. When learning with an undirected feedback graph G over K actions, the online algorithm observes not only the loss of the action chosen in each round, but also the loss of the actions that are adjacent to it in the graph. Two important special cases of this setting are: prediction with expert advice (when G is a clique) and K -armed bandits (when G has no edges). When losses are generated adversarially, the regret in the feedback graph setting with strong observability has been shown to scale with the independence number α of G . Intuitively, denser graphs, which correspond to smaller independence numbers, provide more feedback to the learner, thus enabling a better control on regret. More specifically, the best known upper and lower bounds on the regret after T rounds are O ( √ αT log K ) and Ω ( √ αT ) [3, 4]. It has been known for three decades that this upper bound is tight for α = 1 (the experts case, [9, 10]). When α = K

√

(the bandits case), the lower bound Ω ( KT ) -which has also been known for nearly three decades [7, 8]-was matched by a corresponding upper bound O ( √ KT ) only in 2009 [5]. These results show that in feedback graphs, the logarithmic factor √ log K is necessary (at least) for the α = 1 case, while it must vanish from the minimax regret as α grows from 1 to K , but the current bounds fail to capture this fact. In this work, we prove new upper and lower regret bounds that for the first time account for this vanishing logarithmic factor.

To prove our new upper bound, we use the standard FTRL algorithm run with the q -Tsallis entropy regularizer ( q -FTRL for short). It is well-known [1] that for q = 1 2 this algorithm (run with appropriate loss estimates) achieves regret O ( √ KT ) when α = K (bandits case), while for q → 1 -the same algorithm (without loss estimates) recovers the bound O ( √ T log K ) when α = 1 (experts case). When G contains all self-loops, we show in Theorem 1 that, if q is chosen as a certain function q ( α, K ) , then q ( α, K ) -FTRL, run with standard importance-weighted loss estimates, achieves regret O (√ αT (1 + log( K/α )) ) . This is a strict improvement over the previous bound, and matches the lower bounds for bandits and experts while interpolating the intermediate cases. This interpolation is reflected by our choice of q , which goes from 1 2 to 1 as α ranges from 1 to K . The main technical hurdle in proving this result is an extension to arbitrary values of q ∈ [ 1 2 , 1 ) of a standard resultsee, e.g., [29, Lemma 3]-that bounds in terms of α the variance term in the regret of q -FTRL. In Theorem 2, using a modified loss estimate, this result is extended to any strongly observable undirected graph [2], a class of feedback graphs in which some of the actions do not reveal their loss when played. In Theorem 3, we show via a doubling trick that our new upper bound can also be obtained (up to constant factors) without the need of knowing (or computing) α . As the resulting algorithm is oblivious to α , our analysis also applies to arbitrary sequences of graphs G t , where K is constant but the independence number α t of G t can change over time, and the algorithm observes G t only after choosing an action (the so-called uninformed case). In this setting, the analysis of the doubling trick is complicated by the non-trivial dependence of the regret on the sequence of α t .

We also improve on the Ω ( αT ) lower bound by proving a new Ω (√ αT log α K ) lower bound for all α > 1 . This is the first result showing the necessity-outside the experts case-of a logarithmic factor in the minimax regret for all α < K . Our proof uses a stochastic adversary generating both losses and feedback graphs via i.i.d. draws from a joint distribution. This sequence of losses and feedback graphs can be used to define a hard instance of the multi-task bandits problem, a variant of the combinatorial bandits framework [12]. We then prove our result by adapting known lower bounding techniques for multi-task bandits [6]. Note that for values of α bounded away from 2 and K , the logarithmic factor log α K in the lower bound is smaller than the corresponding factor 1 + log( K/α ) in the upper bound. Closing this gap remains an open problem.

√

1.1 Additional related work

Several previous works have used the q -Tsallis regularizer with q tuned to specific values other than 1 2 and 1 . For example, in [35, Section 4], q is chosen as a function of K to prove a regret bound of O (√ αT (log K ) 3 ) for any strongly observable directed feedback graph, which shaves off a log T factor compared to previous works. This bound is worse than the corresponding bounds for undirected graphs because the directed setting is harder. Specific choices of q have been considered to improve the regret in settings of online learning with standard bandit feedback. For example, the choice q = 2 3 was used in [31] to improve the analysis of regret in bandits with decoupled exploration and exploitation. Regret bounds for arbitrary choices of q are derived in [36, 23] for a best-of-both-worlds analysis of bandits, though q = 1 2 remains the optimal choice. The 1 2 -Tsallis entropy and the Shannon entropy ( q = 1 ) regularizers have been combined before in different ways to obtain best-of-both-worlds guarantees for the graph feedback problem [18, 22]. The idea of using values of q ∈ ( 1 2 , 1) for feedback graphs is quite natural and has been brought up before, e.g., in [32], but achieving an improved dependence on the graph structure by picking a suitable value of q has not been, to the best of our knowledge, successfully pursued before. On the other hand, an approach based on a similar use of the q -Tsallis regularizer has been employed by [26] for the problem of multiarmed bandits with sparse losses to achieve a O (√ sT ln( K/s ) ) regret bound, where s is the maximum number of nonzero losses at any round.

Our lower bound is reminiscent of the Ω (√ KT log K N ) lower bound proved in [33] for the problem of bandits with expert advice (with N ≥ K being the number of experts); see also [17] and [34]. In

that problem, at each time step, experts suggest distributions over actions to the learner, whose regret is computed against the best expert in hindsight. Although the two settings are different, the variant of the multitask bandit problem that our lower bound construction simulates is the same as the one used in the proof of [17, Theorem 7].

2 Problem Setting

̸

For any integer n ≥ 1 , let [ n ] = { 1 , . . . , n } . We consider the following game played over T rounds between a learner with action set V = [ K ] and the environment. At the beginning of the game, the environment secretly selects a sequence of losses ( ℓ t ) t ∈ [ T ] , where ℓ t : V → [0 , 1] , 2 and a sequence of undirected graphs ( G t ) t ∈ [ T ] over the set of actions V , that is, G t = ( V, E t ) . At any time t , the learner selects an arm I t (possibly at random), then pays loss ℓ t ( I t ) and observes the feedback graph G t and all losses ℓ t ( i ) of neighbouring actions i ∈ N G t ( I t ) , where N G t ( i ) = { j ∈ V : ( i, j ) ∈ E t } (see Online Protocol 1). In this work, we only focus on strongly observable graphs [2]. An undirected graph G is strongly observable if for every i ∈ V , at least one of the following holds: i ∈ N G ( i ) or i ∈ N G ( j ) for all j = i .

The performance of the learner is measured by the regret

where the expectation is over the learner's internal randomization.

Online Protocol 1 Online learning with feedback graphs

environment: (hidden) losses ℓ t : V → [0 , 1] and graphs G t = ( V, E t ) , for all t = 1 , . . . , T for t = 1 , . . . , T do

The learner picks an action I t ∈ V (possibly at random)

The learner incurs loss ℓ t ( I t )

The learner observes losses end for

We denote by ∆ K the simplex { p ∈ [0 , 1] K : ∥ p ∥ 1 = 1 } . For any graph G , we define its independence number as the cardinality of the largest set of nodes such that no two nodes are neighbors, and denote it by α ( G ) . For simplicity, we use N t to denote the neighbourhood N G t in the graph G t and we use α t to denote the independence number α ( G t ) of G t at time t .

3 FTRL with Tsallis Entropy for Undirected Feedback Graphs

As a building block, in this section, we focus on the case when all the feedback graphs G 1 , . . . , G T have the same independence number α 1 = · · · = α T = α , whereas the general case is treated in the next section. For simplicity, we start with the assumption that all nodes have self-loops: ( i, i ) ∈ E t for all i ∈ V and all t . We later lift this requirement and show that the regret guarantees that we provide can be extended to general strongly observable undirected feedback graphs, only at the cost of a constant multiplicative factor.

The algorithm we analyze is q -FTRL (described in Algorithm 1), which is an instance of the follow the regularized leader (FTRL) framework-see, e.g., [30, Chapter 7]-with the (negative) q -Tsallis entropy

as the regularizer, whose parameter q ∈ (0 , 1) can be tuned according to our needs. Since we do not observe all the losses in a given round, the algorithm makes use of unbiased estimates for the losses. When all self-loops are present, we define the estimated losses in the following standard manner. Let

{(

i, ℓ

t

(

i

)

)

:

i

∈

N

G

t

(

I

t

)

}

and graph

G

t

I t be the action picked at round t , which is drawn from the distribution p t ∈ ∆ K maintained by the algorithm, the loss estimate for an action i ∈ V at round t is given by

where P t ( i ) = P ( I t ∈ N t ( i ) ) = ∑ j ∈ N t ( i ) p t ( j ) . This estimate is unbiased in the sense that E t [ ̂ ℓ t ( i ) ] = ℓ t ( i ) for all t ∈ [ T ] and all i ∈ V , where we denote E t [ · ] = E [ · | I 1 , . . . , I t -1 ] .

A key part of the standard regret analysis of q -FTRL (see, e.g., the proof of Lemma 3 in Appendix A) is handling the variance term, which, with the choice of estimator given in (1), takes the following form

By Hölder's inequality, this term can be immediately upper bounded by

while previous results on the regret analysis of multiarmed bandits with graph feedback [29, 3] would give

√

However, the former result would only recover a O ( KT ) regret bound (regardless of α ) with the best choice of q = 1 / 2 , which could be trivially achieved by ignoring side-observations of the losses, whereas the latter bound would only manage to achieve a O ( √ αT ln K ) regret bound, incurring the extra √ ln K factor for all values of α . Other results in the literature (e.g., see [2, 4, 16, 19, 22, 25, 32, 35]) do not bring an improvement in this setting when bounding the B t ( q ) term and, hence, do not suffice for achieving the desired regret bound. The following lemma provides a novel and improved bound on quantities of the same form as B t ( q ) in terms of the independence number α t = α of the undirected graph G t .

Lemma1. Let G = ( V, E ) be any undirected graph with | V | = K vertices and independence number α ( G ) = α . Let b ∈ [0 , 1] , p ∈ ∆ K and consider any nonempty subset U ⊆ { v ∈ V : v ∈ N G ( v ) } . Then,

Proof. First of all, observe that we can restrict ourselves to the subgraph G [ U ] induced by U , i.e., the graph G [ U ] = ( U, E ∩ ( U × U )) . This is because the neighbourhoods in this graph are such that N G [ U ] ( v ) ⊆ N G ( v ) for all v ∈ U , and its independence number is α ( G [ U ]) ≤ α ( G ) . Hence, it suffices to prove the claimed inequality for any undirected graph G = ( V, E ) with all self-loops, any p ∈ [0 , 1] K such that ∥ p ∥ 1 ≤ 1 , and the choice U = V . We assume this in what follows without loss of generality.

For any subgraph H ⊆ G with vertices V ( H ) ⊆ V , denote the quantity we want to upper bound by

Our aim is thus to provide an upper bound to Q ( G ) .

Consider a greedy algorithm that incrementally constructs a subset of vertices in the following way: at each step, it selects a vertex v that maximizes p ( v ) b / (∑ u ∈ N G ( v ) p ( u ) ) , it adds v to the solution, and it removes v from G together with its neighbourhood N G ( v ) . This step is iterated on the remaining graph until no vertex is left.

Let S = { v 1 , . . . , v s } ⊆ V be the solution returned by the above greedy algorithm on G . Also let G 1 , . . . , G s +1 be the sequence of graphs induced by the operations of the algorithm, where G 1 = G and G s +1 is the empty graph, and let N r ( v ) = N G r ( v ) for v ∈ V ( G r ) . At every step r ∈ [ s ] of the greedy algorithm, the contribution to Q ( G ) of the removed vertices N r ( v r ) amounts to

where the last inequality is due to the fact that N i ( v ) ⊆ N j ( v ) for all i ≥ j and v ∈ V i . Therefore, we can observe that

The solution S is an independent set of G by construction. Consider now any independent set A ⊆ V of G . We have that

where the second inequality follows by Jensen's inequality and the fact that b ∈ [0 , 1] .

Observe that this upper bound is tight for general probability distributions p ∈ ∆ K over the vertices V of any strongly observable undirected graph G (containing at least one self-loop), as it is exactly achieved by the distribution p ⋆ ∈ ∆ K defined as p ⋆ ( i ) = 1 | S | I { i ∈ S } for some maximum independent set S ⊆ V of G . Using this lemma, the following theorem provides our improved upper bound under the simplifying assumptions we made thus far.

Theorem 1. Let G 1 , . . . , G T be a sequence of undirected feedback graphs, where each G t contains all self-loops and has independence number α t = α for some common value α ∈ [ K ] . If Algorithm 1 is run with input

and loss estimates (1) , then its regret satisfies R T ≤ 2 √ eαT (2 + ln( K/α )) .

Proof. One can verify that for any i ∈ V , the loss estimate ̂ ℓ t ( i ) defined in (1) satisfies E t [ ̂ ℓ t ( i ) 2 ] ≤ 1 /P t ( i ) . Hence, using also that E t [ ̂ ℓ t ( i ) ] = ℓ t ( i ) , Lemma 2 in Appendix A implies that

where the second inequality follows by Lemma 1 with b = 1 -q since all actions i ∈ V are such that i ∈ N G ( i ) . Our choices for q and η allow us to further upper bound the right-hand side of (5) by

The regret bound achieved in the above theorem achieves the optimal regret bound for the experts setting (i.e., α = 1 ) and the bandits setting (i.e., α = K ) simultaneously. Moreover, it interpolates the intermediate cases for α ranging between 1 and K , introducing the multiplicative logarithmic factor only for graphs with independence number strictly smaller than K . We remark that the chosen values of q and η do in fact minimize the right-hand side of (5). Note that we relied on the knowledge of α to tune the parameter q . This is undesirable in general. We will show how to lift this requirement in Section 4. The same comment applies to Theorem 2, below.

We now show how to achieve the improved regret bound of Theorem 1 in the case of strongly observable undirected feedback graphs where some self-loops may be missing; i.e., there may be actions i ∈ V such that i / ∈ N G ( i ) . Using the loss estimator defined in (1) may lead to a large variance term due to the presence of actions without self-loops. One approach to deal with this-see, e.g., [35] or [28]-is to suitably alter the loss estimates of these actions.

Define S t = { i ∈ V : i / ∈ N t ( i ) } as the subset of actions without self-loops in the feedback graph G t at each time step t ∈ [ T ] . The idea is that we need to carefully handle some action i ∈ S t only in the case when the probability p t ( i ) of choosing i at round t is sufficiently large, say, larger than 1 / 2 . Define the set of such actions as J t = { i ∈ S t : p t ( i ) > 1 / 2 } and observe that | J t | ≤ 1 . Similarly to [35], define new loss estimates

for which it still holds that E t [ ̂ ℓ t ] = ℓ t and that E t [ ̂ ℓ t ( i ) 2 ] ≤ 1 /P t ( i ) for all i / ∈ J t . This change, along with the use of Lemma 1 for the actions in V \ S t , suffices in order to prove the following regret bound (see Appendix B for the proof) when the feedback graphs do not necessarily contain self-loops for all actions.

Theorem 2. Let G 1 , . . . , G T be a sequence of strongly observable undirected feedback graphs, where each G t has independence number α t = α for some common value α ∈ [ K ] . If Algorithm 1 is run with input

and loss estimates (6) , then its regret satisfies R T ≤ 6 √ eαT (2 + ln( K/α )) .

4 Adapting to Arbitrary Sequences of Graphs

In the previous section, we assumed for simplicity that all the graphs have the same independence number. This independence number was then used to tune q , the parameter of the Tsallis entropy regularizer used by the algorithm. In this section, we show how to extend our approach to the case when the independence numbers of the graphs are neither the same nor known a-priori by the learner. Had these independence numbers been known a-priori, one approach is to set q as in Theorem 2 but using the average independence number

Doing so would allow us to achieve a O (√ ∑ T t =1 α t (1 + ln( K/α T )) ) regret bound. We now show that we can still recover a bound of the same order without prior knowledge of α T . For round t and any fixed q ∈ [0 , 1] , define

We know from Lemma 1 that H t ( q ) ≤ α q t . Thus, we can leverage these observations and use a doubling trick (similar in principle to [3]) to guess the value of α T . This approach is outlined in Algorithm 2. Starting with r = 0 and T r = 1 , the idea is to instantiate Algorithm 1 at time-step T r with q and η set as in Theorem 2 but with 2 r replacing the independence number. Then, at t ≥ T r , we increment r and restart Algorithm 1 only if

since (again thanks to Lemma 1) the left-hand side of the above inequality is always upper bounded by α T . The following theorem shows that this approach essentially enjoys the same regret bound of Theorem 2 up to an additive log 2 α T term.

Algorithm 2 q -FTRL for an arbitrary sequence of strongly observable undirected graphs

input: Time horizon T

define: For each r ∈ { 0 , . . . , ⌊ log 2 K ⌋} ,

initialization: T 0 ← 1 , r ← 0 , instantiate Algorithm 1 with q = q 0 , η = η 0 , and loss estimates (6) for t = 1 , . . . , T do

Perform one step of the current instance of Algorithm 1

Restart Algorithm 1 with q = q r , η = η r , and loss estimates (6)

end if end for

Theorem 3. Let C = 4 √ 6 e √ π + √ 4 -2 ln 2 ln 2 . Then, the regret of Algorithm 2 satisfies

Proof sketch. For simplicity, we sketch here the proof for the case when in every round t , all the nodes have self-loops; hence, H t ( q ) = B t ( q ) . See the full proof in Appendix C, which treats the general case in a similar manner. Let n = ⌈ log 2 α T ⌉ and assume without loss of generality that α T > 1 . Since Lemma 1 implies that for any r and t , B t ( q r ) ≤ α q r t , we have as a consequence that for any t ≥ T r ,

Hence, the maximum value of r that the algorithm can reach is n -1 . In doing so, we will execute n instances of Algorithm 1, each corresponding to a value of r ∈ { 0 , . . . , n -1 } . For every such r , we upper bound the instantaneous regret at step T r +1 -1 (the step when the restarting condition is satisfied) by 1 , hence the added log 2 α T term in the regret bound. For the rest of the interval; namely, for t ∈ [ T r , T r +1 -2] , we have via (4) that the regret of Algorithm 1 is bounded by

Define T r : r +1 = T r +1 -T r -1 , and notice that

where the first inequality follows due to Jensen's inequality since q r ∈ (0 , 1) , and the second follows from the restarting condition of Algorithm 2. After, plugging this back into (7), we can simply use the definitions of η r and q r and bound the resulting expression in a similar manner to the proof of Theorem 1. Overall, we get that

from which the theorem follows by using Lemma 4 in Appendix A, which shows, roughly speaking, that the sum on the right-hand side is of the same order as its last term.

Although Algorithm 2 requires knowledge of the time horizon, this can be dealt with by applying a standard doubling trick on T at the cost of a larger constant factor. It is also noteworthy that the bound we obtained is of the form √ Tα T (1 + ln( K/α T )) and not √∑ t α t (1 + ln( K/α t )) . Although both coincide with the bound of Theorem 2 when α t is the same for all time steps, the latter is smaller via the concavity of x (1 + ln( K/x )) in x . It is not clear, however, whether there is a tuning of q ∈ (0 , 1) that can achieve the second bound (even with prior knowledge of the entire sequence α 1 , . . . , α T of independence numbers).

5 An Improved Lower Bound via Multitask Learning

In this section we provide a new lower bound on the minimax regret showing that, apart from the bandits case, a logarithmic factor is indeed necessary in general. When the graph is fixed over time, it is known that a lower bound of order √ αT holds for any value of α [3, 29]. Whereas for the experts case ( α = 1 ), the minimax regret is of order 3 √ T ln K [9]. The following theorem provides, for the first time, a lower bound that interpolates between the two aforementioned bounds for the intermediate values of α .

Theorem 4. Pick any K ≥ 2 and any α such that 2 ≤ α ≤ K . Then, for any algorithm and for all T ≥ α log α K 4 log(4 / 3) , there exists a sequence of losses and feedback graphs G 1 , . . . , G T such that α ( G t ) = α for all t = 1 , . . . , T and

In essence, the proof of this theorem (see Appendix D) constructs a sequence of feedback graphs and losses that is equivalent to a hard instance of the multitask bandit problem (MTB) [12], an important special case of combinatorial bandits with a convenient structure for proving lower bounds [6, 15, 21]. We consider a variant of MTB in which, at the beginning of each round, the decision-maker selects an arm to play in each one of M stochastic bandit games. Subsequently, the decision-maker only observes (and suffers) the loss of the arm played in a single randomly selected game. For proving the lower bound, we use a class of stationary stochastic adversaries (i.e., environments), each generating graphs and losses in a manner that simulates an MTB instance.

Fix 2 ≤ α ≤ K = | V | and assume for simplicity that M = log α K is an integer. We now construct an instance of online learning with time-varying feedback graphs G t = ( V, E t ) with α ( G t ) = α that is equivalent to an MTB instance with M bandit games each containing α 'base actions'. Since K = α M , we can uniquely identify each action in V with a vector a = ( a (1) , . . . , a ( M ) ) in

(

1

,

1

,

1)

(

1

,

1

,

2)

(

1

,

2

,

1)

(

1

,

2

,

2)

C

1

,

1

(

2

,

1

,

1)

(

2

,

1

,

2)

(

2

,

2

,

1)

(

2

,

2

,

2)

C

1

,

2

G

1

(1

,

1

,

1)

(1

,

1

,

2)

(2

,

1

,

1)

(2

,

1

,

2)

C

2

,

1

(1

,

2

,

1)

(1

,

2

,

2)

(2

,

2

,

1)

(2

,

2

,

2)

C

2

,

2

G

2

(1

,

1

,

1

)

(1

,

2

,

1

)

(2

,

1

,

1

)

(2

,

2

,

1

)

C

3

,

1

(1

,

1

,

2

)

(1

,

2

,

2

)

(2

,

1

,

2

)

(2

,

2

,

2

)

C

3

,

2

G

3

[ α ] M . The action a t ∈ V chosen by the learner at round t is equivalent to a choice of base actions a t (1) , . . . , a t ( M ) in the M games. The feedback graph at every round is sampled uniformly at random from a set of M undirected graphs { G i } i M =1 , where G i = ( V, E i ) is such that ( a, a ′ ) ∈ E i if and only if a ( i ) = a ′ ( i ) . This means (see Figure 1) that each graph G i consists of α isolated cliques { C i,j } α j =1 such that an action a belongs to clique C i,j if and only if a ( i ) = j . Clearly, the independence number of any such graph is α . Drawing feedback graph G t = G i corresponds to the activation of game i in the MTB instance. Hence, choosing a t ∈ V with feedback graph G t = G i is equivalent to playing base action a t ( i ) in game i in the MTB. As for the losses, we enforce that, given a feedback graph G t , all actions that belong to the same clique of the feedback graph are assigned the same loss. Namely, if G t = G i and a ( i ) = a ′ ( i ) = j , then ℓ t ( a ) = ℓ t ( a ′ ) , which can be seen as the loss ℓ t ( j ) assigned to base action j in game G i . To choose the distribution of the losses for the base actions, we apply the classic needle-in-a-haystack approach of [7] over the M games. More precisely, we construct a different environment for each action a ∈ V in such a way that the distribution of the losses in each MTB game slightly favors (with a difference of a small ε > 0 ) the base action corresponding to a in that game. The proof then proceeds similarly to, for example, the proof of Theorem 5 in [6] or Theorem 7 in [17].

While both our upper and lower bounds achieve the desired goal of interpolating between the minimax rates of experts and bandits, the logarithmic factors in the two bounds are not exactly matching. In particular, if we compare 1 + log 2 ( K/α ) and log α K , we can see that although they coincide at α = 2 and α = K , the former is larger for intermediate values. It is reasonable to believe that the upper bound is of the correct order, seeing as it arose naturally as a result of choosing the best parameter for the Tsallis entropy regularizer, whereas achieving the extra logarithmic term in the lower bound required a somewhat contrived construction.

6 Conclusions and Future Work

In this work, we have shown that a proper tuning of the q -FTRL algorithm allows one to achieve a O (√ αT (1 + ln( K/α )) ) regret for the problem of online learning with undirected strongly observable feedback graphs. Our bound interpolates between the minimax regret rates of the bandits and the experts problems, the two extremes of the strongly observable graph feedback spectrum.

Furthermore, we have shown that an analogous bound can be achieved when the graphs vary over time, and without requiring any prior knowledge on the graphs. These results are complemented by our new lower bound of Ω (√ αT (ln K ) / (ln α ) ) , which holds for α ≥ 2 and shows the necessity of a logarithmic factor in the minimax regret except for the bandits case. While our results provide the tightest characterization to date of the minimax rate for this setting, closing the small remaining gap (likely on the lower bound side) is an interesting problem. After the submission of this manuscript, a subsequent work [14] showed a lower bound for fixed feedback graphs composed of disjoint cliques that would imply worst-case optimality (up to constant factors) of our proposed algorithm for each pair of K and α -see Appendix E for a more detailed comparison with results therein. Extending our results to the case of directed strongly observable feedback graphs is a considerably harder task-see Appendix F for a preliminary discussion. Better understanding this more general setting is an interesting future direction.

References

J. D. Abernethy, C. Lee, and A. Tewari. Fighting bandits with a new kind of smoothness. Advances in Neural Information Processing Systems , 28, 2015.

N. Alon, N. Cesa-Bianchi, O. Dekel, and T. Koren. Online learning with feedback graphs: Beyond bandits. In P. Grünwald, E. Hazan, and S. Kale, editors, Proceedings of the 28th Conference on Learning Theory , volume 40 of JMLR Workshop and Conference Proceedings , pages 23-35. JMLR.org, 2015.

N. Alon, N. Cesa-Bianchi, C. Gentile, S. Mannor, Y. Mansour, and O. Shamir. Nonstochastic multi-armed bandits with graph-structured feedback. SIAM Journal on Computing , 46(6):17851826, 2017.

N. Alon, N. Cesa-Bianchi, C. Gentile, and Y. Mansour. From bandits to experts: A tale of domination and independence. Advances in Neural Information Processing Systems , 26, 2013.

J.-Y. Audibert and S. Bubeck. Minimax policies for adversarial and stochastic bandits. In COLT , volume 7, pages 1-122, 2009.

J.-Y. Audibert, S. Bubeck, and G. Lugosi. Regret in online combinatorial optimization. Mathematics of Operations Research , 39(1):31-45, 2014.

P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. Gambling in a rigged casino: The adversarial multi-armed bandit problem. In Proceedings of IEEE 36th annual foundations of computer science , pages 322-331. IEEE, 1995.

P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002.

N. Cesa-Bianchi, Y. Freund, D. Haussler, D. P. Helmbold, R. E. Schapire, and M. K. Warmuth. How to use expert advice. Journal of the ACM (JACM) , 44(3):427-485, 1997.

N. Cesa-Bianchi, Y. Freund, D. P. Helmbold, D. Haussler, R. E. Schapire, and M. K. Warmuth. How to use expert advice. In Proceedings of the 25th Annual ACM Symposium on Theory of Computing , pages 382-391, 1993.

N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games . Cambridge University Press, 2006.

N. Cesa-Bianchi and G. Lugosi. Combinatorial bandits. Journal of Computer and System Sciences , 78(5):1404-1422, 2012.

S.-H. Chang, P. C. Cosman, and L. B. Milstein. Chernoff-type bounds for the Gaussian error function. IEEE Transactions on Communications , 59(11):2939-2944, 2011.

H. Chen, Y. He, and C. Zhang. On interpolating experts and multi-armed bandits. arXiv preprint arXiv:2307.07264 , 2023.

A. Cohen, T. Hazan, and T. Koren. Tight bounds for bandit combinatorial optimization. In Conference on Learning Theory , pages 629-642. PMLR, 2017.

C. Dann, C. Wei, and J. Zimmert. A blackbox approach to best of both worlds in bandits and beyond. In The Thirty Sixth Annual Conference on Learning Theory , volume 195 of Proceedings of Machine Learning Research , pages 5503-5570. PMLR, 2023.

K. Eldowa, N. Cesa-Bianchi, A. M. Metelli, and M. Restelli. Information-theoretic regret bounds for bandits with fixed expert advice. In IEEE Information Theory Workshop , pages 30-35. IEEE, 2023.

L. Erez and T. Koren. Towards best-of-all-worlds online learning with feedback graphs. Advances in Neural Information Processing Systems , 34:28511-28521, 2021.

E. Esposito, F. Fusco, D. van der Hoeven, and N. Cesa-Bianchi. Learning on the edge: Online learning with stochastic feedback graphs. In Advances in Neural Information Processing Systems , volume 35, pages 34776-34788, 2022.

D. Haussler, J. Kivinen, and M. Warmuth. Sequential prediction of individual sequences under general loss functions. IEEE Transactions on Information Theory , 44(5):1906-1925, 1998.

S. Ito, D. Hatano, H. Sumita, K. Takemura, T. Fukunaga, N. Kakimura, and K.-I. Kawarabayashi. Improved regret bounds for bandit combinatorial optimization. Advances in Neural Information Processing Systems , 32, 2019.

S. Ito, T. Tsuchiya, and J. Honda. Nearly optimal best-of-both-worlds algorithms for online learning with feedback graphs. Advances in Neural Information Processing Systems , 35:2863128643, 2022.

T. Jin, J. Liu, and H. Luo. Improved best-of-both-worlds guarantees for multi-armed bandits: FTRL with general regularizers and multiple optimal arms. arXiv preprint arXiv:2302.13534 , 2023.

R. M. Karp. Reducibility among combinatorial problems. In R. E. Miller and J. W. Thatcher, editors, Proceedings of a Symposium on the Complexity of Computer Computations , pages 85-103. Plenum Press, 1972.

T. Kocák, G. Neu, M. Valko, and R. Munos. Efficient learning by implicit exploration in bandit problems with side observations. In Advances in Neural Information Processing Systems , volume 27, pages 613-621, 2014.

J. Kwon and V. Perchet. Gains and losses are fundamentally different in regret minimization: The sparse case. Journal of Machine Learning Research , 17(227):1-32, 2016.

T. Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

H. Luo, H. Tong, M. Zhang, and Y. Zhang. Improved high-probability regret for adversarial bandits with time-varying feedback graphs. In International Conference on Algorithmic Learning Theory , pages 1074-1100. PMLR, 2023.

S. Mannor and O. Shamir. From bandits to experts: On the value of side-observations. Advances in Neural Information Processing Systems , 24, 2011.

F. Orabona. A modern introduction to online learning. CoRR , abs/1912.13213, 2019.

C. Rouyer and Y. Seldin. Tsallis-inf for decoupled exploration and exploitation in multi-armed bandits. In Conference on Learning Theory , pages 3227-3249. PMLR, 2020.

C. Rouyer, D. van der Hoeven, N. Cesa-Bianchi, and Y. Seldin. A near-optimal best-of-bothworlds algorithm for online learning with feedback graphs. In Advances in Neural Information Processing Systems , 2022.

Y. Seldin and G. Lugosi. A lower bound for multi-armed bandits with expert advice. In 13th European Workshop on Reinforcement Learning (EWRL) , volume 2, page 7, 2016.

N. M. Vural, H. Gokcesu, K. Gokcesu, and S. S. Kozat. Minimax optimal algorithms for adversarial bandit problem with multiple plays. IEEE Transactions on Signal Processing , 67(16):4383-4398, 2019.

J. Zimmert and T. Lattimore. Connections between mirror descent, Thompson sampling and the information ratio. Advances in Neural Information Processing Systems , 32, 2019.

J. Zimmert and Y. Seldin. Tsallis-inf: An optimal algorithm for stochastic and adversarial bandits. The Journal of Machine Learning Research , 22(1):1310-1358, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a classic problem of online learning with feedback graphs, which interpolates the full-information feedback and the bandit feedback. Specifically, the authors consider the case where the feedback graph is undirected, meaning that if node $i$ can observe node $j$, then node $j$ can observe node $i$. The best known $T$-dependent upper bound for this problem is $O(\\sqrt{\\alpha T\\ln K})$ by [Alon et al., 2013]. For general strongly-observable graph, the best known $T$-dependent upper bound is $O(\\sqrt{\\alpha T\\ln^3 K})$ proposed in [Zimmert and Lattimore, 2019]. In this paper, the authors propose an FTRL algorithm with q-Tsallis entropy regularization, showing $O(\\sqrt{\\alpha T(1+\\ln \\frac{K}{\\alpha})})$ upper bound. This recovers the minimax regret bound in both bandit $O(\\sqrt{KT})$ regret and full-information setup $O(\\sqrt{T\\ln K})$. Moreover, the authors also propose an improved $\\Omega(\\sqrt{\\alpha T\\frac{\\ln K}{\\ln \\alpha}})$ lower bound compared with the $\\Omega(\\sqrt{\\alpha T})$ lower bound proven in [Alon et al., 2015], though this lower bound does not match the upper bound obtained by the q-Tsallis FTRL algorithm. In addition, the authors also generalize their results to the undirect strongly observable graphs and time-varying graphs.\n\nSTRENGTHS:\n- The paper is well-written and the proposed algorithm is easy to follow.\n- The designed algorithm improves upon the best known $T$-dependent regret bound for undirected strongly observable graphs. Specifically, the important technical contribution is Lemma 1, which shows that the stability term of the FTRL can be bounded by $\\alpha^{1+b}$ where $b$ is decided by the parameter choice of Tsallis-entropy, compared to the one with log factors obtained in [Alon et al., 2015] and [Zimmert and Lattimore, 2019]. \n- The lower bound also tries to bridge the previous gap between the full-information case and the feedback graph case, although there is still a gap between the upper and the lower bound and the technical used in the lower bound is very similar to the one used for proving the lower bound for contextual bandit in [Seldin and Lugosi, 2016].\n\nWEAKNESSES:\nI do not find major weaknesses of this paper, except that the current upper bound is only obtained for the undirected strongly observable graph, which is also discussed by the authors in the appendix. Generalizing this technique to the general strongly observable graphs would be interesting.\n\nQUESTIONS:\n- The current algorithm is an FTRL-based algorithm. I wonder whether a certain type of online mirror descent based algorithm with a time-varying learning rate can also achieve similar results?\n\nLIMITATIONS:\nSee Weakness and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates no-regret online learning algorithms performing under the feedback graph model. More precisely, at each round $t$ a learner selects an action (out of set of possible actions) incurring the cost associated with the specific action at the specific round. The actions are additionally vertices of an undirected graph that changes from round to round. Once the action $i$ is selected at round $t$, the learner is also informed on the costs of the neighboring actions at the specific rounds. The latter setting generalizes the bandit setting (the respective graph is composed by isolated vertices) as well as the full-information setting (the respective graph is a clique). Initially the authors consider the special case where every graph at each round admits independence number $\\alpha$ ($\\alpha =1$ for cliques and $\\alpha =K$ for graphs with $K$ isolated edges). The authors first provide a nearly tight regret guarantee ($O(\\sqrt{\\alpha T \\log (k/\\alpha)})$) for the above special case. The authors also provide a nearly matching $\\Omega(\\sqrt{\\alpha T \\log K / \\log \\alpha})$ lower bound on the regret. Finally the authors remove the assumption that all the arriving graphs admit the same independence number and provide an $O(\\sqrt{\\overline{\\alpha} T \\log (k/\\overline{\\alpha})})$ regret bound where $\\overline{a}$ is the time-average independence number of the graphs.\n\nSTRENGTHS:\nI think the paper is very interesting both from the perspective of results as well as from the perspective of techniques. I find very interesting the fact that the provided regret bound matches the respective regret bounds for the bandit ($\\alpha = K$) and the full-information case ($\\alpha=1$).  Also the authors provide a nearly matching lower bound that is based on a novel reduction to multitask learning. Both the techniques for the upper and lower bound require no trivial ideas that are nicely illustrated in the current write up. I also liked the fact that the provided algorithm can be modified to handle the case of time-varying independence numbers.\n\nI overall believe that the paper provides very solid results and that will be of great interest of the online learning audience of NeurIPS.\n\nWEAKNESSES:\nThe only limitation of the paper that I can find is the assumption on undirected graphs. The latter is also signified by the authors and considering the complexity of the current I think it is more than fair to consider the directed case for future work.\n\nQUESTIONS:\n-\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTwo classical learning problems are learning with experts and multi-armed bandits. In both problems, a learner interacts during $T$ rounds with a set of $K$ actions by selecting an action at each round. After each round, the loss/reward at that step is revealed for all actions (resp. only for the selected action) in the expert (resp. bandit) problem. Mannor and Shamir [24] provided a convenient interpolation between the two problems by considering a generalization via feedback graphs. At each step $t$, losses are observed only at neighbor actions to the selected one, according to a graph $G_t$. The expert case corresponds to the clique, while the bandit case corresponds to graphs without edges. In this context, it is known that if the graphs have independence number $\\alpha$, the minimax regret lies between $O(\\sqrt{\\alpha T\\log K})$ and $\\Omega(\\sqrt{\\alpha T})$. Further, the lower (resp. upper) bound is tight for learning with experts (resp. multi-armed bandits) for which $O(\\sqrt{KT})$ regret is achievable (resp. a lower bound $\\Omega(\\sqrt{KT\\log K})$ is known).\n\n\nThis paper investigates the corresponding $\\sqrt{\\log K}$ gap between the case $\\alpha=1$ (clique/experts) and $\\alpha=K$ (multi-armed bandits) and gives interpolating upper bound $O(\\sqrt{\\alpha T(1+\\log K/\\alpha)})$ on the regret provided that the graphs are undirected and strongly observable (each vertex has either a self-loop or neighbor to all other actions), with the same algorithm---FTRL with $q$-Tsallis entropy---run with appropriate parameter $q(\\alpha)$ and loss estimates. Via a doubling trick, they show that one can achieve this bound up to an additive $\\log \\alpha$ term without prior knowledge on $\\alpha$ (and can be further extended to the case when graphs $G_t$ do not share the same independence number).\n\nLast, the authors provide an improved interpolating lower bound $\\Omega(\\sqrt{\\alpha T \\log K/\\log \\alpha})$, which requires the graphs $G_t$ to vary over time (but with same independence number), showing that when $\\alpha$ is sub-polynomial in $K$, learning with feedback graphs requires an extra factor compared to the known lower bound $\\Omega(\\sqrt{\\alpha T})$. This is shown via a reduction to the multitask bandit problem (different tasks are simulated by changing the graph over time).\n\nSTRENGTHS:\nThe paper is very well-written and pleasant to read. The question is well posed, and motivated by the fact that improving the $\\sqrt{\\log K}$ factor for learning with experts, from $O(\\sqrt{KT\\log K})$ to $O(\\sqrt{KT})$ remained open for a significant amount of time in the literature. Hence, interpolating the regret bounds in terms of the remaining factor $\\sqrt{\\log K}$ between the known bounds $O(\\sqrt{\\alpha T\\log k})$ and $\\Omega(\\sqrt{\\alpha T})$ seems an important question.\n\nThe upper bounds use the same algorithm (FTRL with $q$-Tsallis entropy) with adapted parameters $q$, which also interpolates between known approaches for the extreme cases $\\alpha=1$ or $\\alpha=K$, giving new insights on the impact of this parameter on the learning behavior.\n\nWEAKNESSES:\nAlthough the paper presents new contributions to interpolate between experts and bandits, the proofs follow classical arguments in the literature and seem to bring limited original ideas. In particular, it was known that the $q$-Tsallis entropy specifically allows achieving the minimax entropy in both end cases. The main difficulty in the generalization to feedback graphs is in a novel bound for the variance term in the FTRL analysis (Lemma 3), which itself is a generalization of known results in the literature for $q=1$. The rest of the proof follows standard analysis.\n\nThe lower bound heavily relies on the fact that graphs are allowed to vary over time. It seems that the approach cannot be extended to the important case of interest of a fixed graph (which seemed to be the main case studied in the previous literature on feedback graph bandits). The paper would also be significantly strengthened if the lower bound could be improved to match the upper bound when e.g. the independence number $\\alpha$ grows as a polynomial in $K$---in that case, the present paper does not provide improvements in terms of rates compared to the existing literature. As such, the gap left by this paper seems to remain quite important.\n\nQUESTIONS:\n- Does the analysis carry over for adaptive adversaries, or is the oblivious assumption important?\n- l.97 the independence number is the cardinality of such a set, right?\n- What happens if we assume that the graph is constant over time? Would you expect similar lower bounds to hold?\n\nLIMITATIONS:\nThe limitations were very well addressed by the authors, who clearly identify the new bound on the variance (Lemma 3) as the key added technical contribution for the upper bounds.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the online learning under partial observations given by an underlying graph structure. This model is a common generalization of the bandit model (where the feed graph just contains self loops) and the full information model (where the graph is a complete graph with self loops). This model has been considered in the literature where it has been shown that the regret (under a technical condition known as strong observability) is $\\tilde{\\Theta} ( \\sqrt{ \\alpha T } )$ where $T$ is the time horizon and $\\alpha$ is the independence number of the graph. Prior to the present paper there was a logarithmic gap in the regret upper and lower bounds. In particular, the previous bound when instantiated in the case of bandits led to an extra $log K$ factor. The present paper addresses this gap by designing an algorithm  that achieves regret $\\sqrt{ \\alpha T (1 + \\log (K/\\alpha)) }$ which gives the optimal bounds in both extremes. Furthermore, the paper presents lower bounds matching their given bounds. \n\nThe algorithm that the paper analyses is a version of FTRL with the Tsallis entropy regularizer. Since the choice of $q=1/2$ is known to be optimal for bandits and $q=1$ is optimal for full information, the natural idea is to interpolate between using a $\\alpha$ dependent choice of $q$. The paper analyses the variance of the natural inverse probability weighted estimator of the loss in a novel way to arrive at the optimal choice to $q$ to get the regret bound. \n\nThe paper extends their analysis to a slightly more general settings (than just graphs with self loops; time varying independence numbers).\n\nSTRENGTHS:\nAs stated in the summary, the main technical novelty is the analysis of the variance of the IPW estimator in the setting of feedback graphs and FTRL. This is itself a rather interesting contribution and I expect this to be useful in various other contexts. The main result presented in the paper resolves an interesting and natural question in the literature.\n\nWEAKNESSES:\nOne main drawback of the approach of $\\alpha$ dependent choice of the $q$ is that it makes the algorithm extremely inefficient (in theory at least). In particular, the independence number of a graph is known to (extremely) hard to approximate; in particular, unless $P=NP$ there is no \"non-trivial\" approximation algorithm even. This is to be contrasted with the previous work in the feedback graph setting which are efficient ( polynomial in $N$ the size of the graph/number of bandits). It remains an excellent open question to see if there is an efficient algorithm achieving this bound.\n\nQUESTIONS:\n- Is it immediately clear how the regret behaves when one only has an estimate of the independence number instead of the exact quantity?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}