Towards Best-of-All-Worlds Online Learning with Feedback Graphs

√ θ ( ) ) ) with adversarial losses ; O ( θ ( ) polylog ) ) with stochastic losses ; andO ( θ ( ) polylog ) + √ θ ( ) ) with stochastic losses subject to adversarial corruptions . Here , θ ( ) is the clique covering number of the graph . Our algorithm is an instantiation of Follow-theRegularized-Leader with a novel regularization that can be seen as a product of a Tsallis entropy component ( inspired by Zimmert and Seldin [ 27 ] ) and a Shannon entropy component ( analyzed in the corrupted stochastic case by Amir et al . [ 3 ] ) , thus subtly interpolating between the two forms of entropies . One of our key technical contributions is in establishing the convexity of this regularizer and controlling its inverse Hessian , despite its complex product structure . 1 Introduction . Online learning models a repeated interaction between a learner and an environment . In this framework , the learner has to choose an arm ( or action ) from a set of # arms , iteratively across ) rounds . After choosing an arm the learner incurs an associated loss and receives feedback from the environment . The feedback the learner receives can range between the full-information setting where after choosing an action the learner sees the losses of all # arms , and themulti-armed bandit ( MAB ) setting where only the loss of the arm chosen at the current round is revealed . Traditionally , the way the losses are generated is either in an adversarial manner ( i.e. , can be arbitrary in each round ) , or stochastically ( i.e. , sampled i.i.d . from an unknown distribution ) . Recently , online algorithms achieving best-of-both-worlds guarantees have drawn significant attention : these are algorithms that are able to perform optimally in both the stochastic and adversarial settings , without prior knowledge of the regime and its parameters . Recent advances in this line of work [ 3 , 27 , 13 , 28 , 18 ] have resulted with algorithms that achieve a remarkable best-of-all-worlds guarantee : in a general intermediate adversarially-corrupted regime , where losses are generated stochastically but then may be modified by an adversary in an arbitrary way , these algorithms attain regret that grows with the square-root of the total amount of corruption introduced . Even more , these guarantees significantly outperform those obtained by specialized algorithms designed exclusively for the corrupted stochastic setting [ 22 , 11 , 23 ] . Our goal in this work is to extend these best-of-all-worlds results from the full-information andMAB settings to more general online learning problems and feedback models . We focus on the general framework of online learning with graph-structure feedback , originally introduced by Mannor and Shamir [ 24 ] and studied extensively since ( e.g. , [ 1 , 10 , 16 , 9 , 21 , 20 , 12 ] ) . In this model , feedback is described by an undirected graph over the arms ; the set of observations received following each prediction is comprised of the loss of the chosen arm as well as the loss of all of its neighbors in . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The feedback graphs framework nicely interpolates between online learning with full-information and learning with bandit feedback , and generalizes to settings with more complex side-observation structure . Full-information online learning is captured as an extreme case where the graph is the clique over the # arms ; on the other extreme , an empty feedback graph that contains no edges corresponds to the multi-armed bandit problem . The optimal regret in the feedback graphs setting is well understood , both in the adversarial and stochastic regimes . In the former , the state-of-the-art is obtained by Alon et al . [ 1 ] who gave an algorithm with regret O ( √ α ( ) ) ) , where α ( ) is the independence number of , and also established its optimality up to log factors . In the latter regime , Cohen et al . [ 10 ] proposed an algorithm whose regret is roughly O ( α ( ) log ) ) , which is again nearly optimal . However , it remains unclear whether there exists a single algorithm that attains both bounds simultaneously , without prior knowledge of the regime , considering that the existing optimal algorithms in the two settings are inherently different . Of course , one can always ignore the additional feedback and use an existing best-of-all-worlds algorithm for MAB ; however , this would result with a direct dependence on the number of arms # . The challenge is then to improve this dependence by exploiting the structure of the feedback graph . 1.1 Our contributions . We make a significant step towards obtaining a best-of-all-worlds guarantee in the feedback graphs model , by giving the first algorithm in this setting that achieves a uniform best-of-all-worlds guarantee and improves over the existing MAB bounds in a non-trivial way . Our algorithm achieves regret roughly of the form θ ( ) polylog ( ) ) when the losses are stochastic i.i.d. , and Õ ( √ θ ( ) ) ) when the losses are fully adversarial ; here θ ( ) denotes the clique covering number of the graph .1 Furthermore , in an intermediate adversarially-corrupted stochastic setting , where the losses are generated stochastically but then may be corrupted by an adversary , we obtain a bound roughly of the form θ ( ) polylog ( ) ) + Õ ( √ θ ( ) ) with being the total amount of corruption , that smoothly interpolates between these two extremes . Crucially , the algorithm achieves the aforementioned bounds simultaneously and agnostically , in the sense that it need not be aware of the actual underlying loss generation process , and in particular , of the actual level of adversarial corruption introduced . To state our results more concretely , let+1 , . . . , +θ be a minimum clique cover of ( here θ = θ ( ) ) , and denote by ∆ : the minimal gap of a suboptimal arm in + : ; namely , the difference in mean losses ( with respect to the stochastic uncorrupted losses ) between an arm in+ : and the best armwhosemean loss is minimal . Finally , define / = ∑ : :∆ : > 0 1/∆ : ( this is the quantity governing the complexity of the learning problem in the purely stochastic case ; see [ 10 ] ) . Then , our main result can be stated as : Theorem ( main , informal ) . There exists an algorithm ( see Algorithm 1 in Section 4 ) which , given a clique covering +1 , . . . , +θ of , obtains an expected regret bound of Õ ( min { √ θ ) , / + √ / } ) in the -corrupted stochastic setting . In particular , in the stochastic setting ( = 0 ) the expected regret of the algorithm is at most Õ ( ∑ : :∆ : > 0 1/∆ : ) , while in the adversarial setting ( = ) ) its expected regret is bounded by Õ ( √ θ ) ) . The above results leverage the graph structure in a non-trivial way and improve the direct dependence on the number of arms # in existing best-of-all-worlds results to a dependence on θ ( ) , which is potentially significantly smaller than # . On the other hand , our bounds do not feature the optimal dependence on the independence number α ( ) ; we leave it as a main open question whether it is possible to obtain a best-of-all-worlds result with α ( ) replacing θ ( ) in the bounds . We remark however , as we discuss in more detail below , that obtaining bounds scaling with the weaker θ ( ) is already a highly non-trivial task and so we believe that our results form a significant step towards obtaining optimal best-of-all-worlds bounds . Another question we leave open is whether our dependence on ) can be improved toO ( log ) ) andO ( √ ) ) in the stochastic and adversarial cases respectively , removing a few excess logarithmic factors in our bounds . We note that the algorithm achieving the bound above is not adaptive to the best clique covering of the feedback graph for which the quantity / above is minimal . In other words , the algorithm requires a specific minimum clique covering as input , and the regret bounds we show take into account the suboptimality gaps with respect to this given covering , which may be worse than other coverings 1The clique covering number of a graph is the minimum number of cliques that cover its set of vertices ; without loss of generality , we can assume that the latter set of cliques is a partition . with respect to these gaps . Improving the algorithm such that it becomes adaptive to the minimum clique covering could be an interesting direction for future research . 1.2 Overview of main ideas and techniques . Our approach is inspired by recent progress in obtaining best-of-all-worlds guarantees in the two extreme cases of our problem : multi-armed bandits [ 27 ] and full-information experts [ 3 ] . Somewhat surprisingly , it has been shown in both cases that this type of guarantee can be obtained by simple algorithms based on the canonical Follow-the-Regularized-Leader template , traditionally used exclusively in adversarial online learning . Our primary technical challenge therefore boils down to designing a convex regularizer given the feedback graph , that carefully interpolates between the well-understood regularization schemes in the full-information and bandit cases . In the full-information case , best-of-all-worlds results [ 3 ] rely on ( a time-varying version of ) the negative Shannon entropy ' ( ? ) = ∑ # 8=1 ? 8 log ? 8 as regularization . In a nutshell , its crucial properties are an O ( log # ) uniform upper bound on its magnitude over the probability simplex , and on the other hand , that its Hessian ∇2 ' ( ? ) = diag ( ? −11 , . . . , ? −1 # ) satisfies ` T ( ∇2 ' ( ? ) ) −1 ` = O ( 1 ) for any loss vector ` ∈ [ 0 , 1 ] # . ( We omit more technical details on third-order conditions and selfbounding properties from this informal discussion . ) In the bandit case , on the other hand , the best-of-all-worlds result of [ 27 ] relies on a Tsallis entropy ( with parameter α = 1/2 ) regularizer ' ( ? ) = − ∑ # 8=1 √ ? 8 , originally proposed in the context of MAB by [ 4 ] . This form of entropy is bounded uniformly over the simplex byO ( √ # ) , and its Hessian ∇2 ' ( ? ) = 14 diag ( ? −3/2 1 , . . . , ? −3/2 # ) satisfies E [ ˜̀T ( ∇2 ' ( ? ) ) −1 ˜̀ ] = O ( √ # ) for loss estimators ˜̀ used in MAB . In both cases , the regularizer strikes the “ right ” balance between the two bounds , which is necessary for optimal regret . Moving to the more complex feedback model induced by a graph , it is natural to expect that an appropriate regularization would emerge as a certain combination of the Shannon and Tsallis entropies . For simplicity , consider a simple graph formed as a union of θ disjoint cliques+1 , . . . , +θ , and let ? be a probability vector over its vertices . Then , we are looking for a regularization that would treat the marginal clique probabilities @ : = ∑ 8∈+ : ? 8 as a Tsallis entropy , while behaving within a clique ( with respect to the internal conditional probabilities ? 8/ @ : for 8 ∈ + : ) like the negative Shannon entropy . Inspecting the technical conditions more deeply , it turns out that we seek a regularizer ' such that the Hessian at ? is lower bounded by a diagonal matrix with the 8 ’ th diagonal entry corresponding to 8 ∈ + : is ? −18 @ −1/2 : . A natural candidate would be simply the sum of the Shannon and Tsallis entropies ; unfortunately , this regularization lacks the bi-level structure we need , and indeed , its Hessian does not admit the desired lower bound . Our main technical innovation is the design of a new convex regularizer that admits the aforementioned lower bound on its Hessian , and on the other hand , is bounded by O ( √ θ log # ) over the simplex . Interestingly , we find that the necessary conditions are met for a regularizer produced by the product of a negative Tsallis entropy ( over the marginals @ : ) and a negative Shannon entropy ( over the conditional ? 8/ @ : ) , rather than their sum . Of course , such a product need not be a convex function in general , and indeed , the regularization we just described fails to be convex even in simple cases ( see Fig . 1 ) . Nevertheless , it turns out that there is a simple fix that makes this bi-level regularization a convex function : a simple linear shift to the Shannon entropy component . Not only that , but the resulting regularizer also admits the stronger Hessian lower bound we require . We refer to this new regularizer by Tsallis-Shannon entropy , and discuss the details of its derivation in Section 3 ; the remainder of the development takes more standard lines and is described in Sections 4 and 5 .

Summary: This paper considers the problem of online learning with feedback graphs. Feedback graphs tell which losses the learner gets to observe each time an action is chosen.  In particular, special cases include the bandit setting when the graph has no edges, and the full information setting when the feedback graph is a clique.  The authors derive bounds that scale in terms of the clique covering number of the graph. This clique covering number is the number of cliques (fully connected subgraphs) that are appearing in the graph. This number is useful because each time one arm is picked, the learner can observe the losses of all the other arms that are in the same clique.  Reasoning about the problem in terms of cliques is interesting because the exploration vs exploitation trade-off is not the same everywhere on the graph. If we consider each clique as one entity, we have a standard multi-armed bandits exploration vs exploitation trade-off. But then, when considering which action to play within the clique, this is a full information problem as we will get the feedback for all actions within the clique either way.  The authors then provide a FTRL algorithm that combines the Tsallis entropy (optimal for MAB) and the Shannon entropy (optimal for Full info).  They also mix in a log barrier regularizer to help with the stability of the algorithm.   The authors derive a single bound that covers simultaneously the adversarial, the stochastic, and the corrupted setting. That bound is near-optimal because of the consideration of the covering clique number rather than the independence number (which basically means that the edges in the feedback graph that are not part of a clique are not taken advantage of), and because the extra log barrier regularization term leads to extra log factors.