Towards Best-of-All-Worlds Online Learning with Feedback Graphs

√ θ ( ) ) ) with adversarial losses ; O ( θ ( ) polylog ) ) with stochastic losses ; andO ( θ ( ) polylog ) + √ θ ( ) ) with stochastic losses subject to adversarial corruptions . Here , θ ( ) is the clique covering number of the graph . Our algorithm is an instantiation of Follow-theRegularized-Leader with a novel regularization that can be seen as a product of a Tsallis entropy component ( inspired by Zimmert and Seldin [ 27 ] ) and a Shannon entropy component ( analyzed in the corrupted stochastic case by Amir et al . [ 3 ] ) , thus subtly interpolating between the two forms of entropies . One of our key technical contributions is in establishing the convexity of this regularizer and controlling its inverse Hessian , despite its complex product structure . 1 Introduction . Online learning models a repeated interaction between a learner and an environment . In this framework , the learner has to choose an arm ( or action ) from a set of # arms , iteratively across ) rounds . After choosing an arm the learner incurs an associated loss and receives feedback from the environment . The feedback the learner receives can range between the full-information setting where after choosing an action the learner sees the losses of all # arms , and themulti-armed bandit ( MAB ) setting where only the loss of the arm chosen at the current round is revealed . Traditionally , the way the losses are generated is either in an adversarial manner ( i.e. , can be arbitrary in each round ) , or stochastically ( i.e. , sampled i.i.d . from an unknown distribution ) . Recently , online algorithms achieving best-of-both-worlds guarantees have drawn significant attention : these are algorithms that are able to perform optimally in both the stochastic and adversarial settings , without prior knowledge of the regime and its parameters . Recent advances in this line of work [ 3 , 27 , 13 , 28 , 18 ] have resulted with algorithms that achieve a remarkable best-of-all-worlds guarantee : in a general intermediate adversarially-corrupted regime , where losses are generated stochastically but then may be modified by an adversary in an arbitrary way , these algorithms attain regret that grows with the square-root of the total amount of corruption introduced . Even more , these guarantees significantly outperform those obtained by specialized algorithms designed exclusively for the corrupted stochastic setting [ 22 , 11 , 23 ] . Our goal in this work is to extend these best-of-all-worlds results from the full-information andMAB settings to more general online learning problems and feedback models . We focus on the general framework of online learning with graph-structure feedback , originally introduced by Mannor and Shamir [ 24 ] and studied extensively since ( e.g. , [ 1 , 10 , 16 , 9 , 21 , 20 , 12 ] ) . In this model , feedback is described by an undirected graph over the arms ; the set of observations received following each prediction is comprised of the loss of the chosen arm as well as the loss of all of its neighbors in . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The feedback graphs framework nicely interpolates between online learning with full-information and learning with bandit feedback , and generalizes to settings with more complex side-observation structure . Full-information online learning is captured as an extreme case where the graph is the clique over the # arms ; on the other extreme , an empty feedback graph that contains no edges corresponds to the multi-armed bandit problem . The optimal regret in the feedback graphs setting is well understood , both in the adversarial and stochastic regimes . In the former , the state-of-the-art is obtained by Alon et al . [ 1 ] who gave an algorithm with regret O ( √ α ( ) ) ) , where α ( ) is the independence number of , and also established its optimality up to log factors . In the latter regime , Cohen et al . [ 10 ] proposed an algorithm whose regret is roughly O ( α ( ) log ) ) , which is again nearly optimal . However , it remains unclear whether there exists a single algorithm that attains both bounds simultaneously , without prior knowledge of the regime , considering that the existing optimal algorithms in the two settings are inherently different . Of course , one can always ignore the additional feedback and use an existing best-of-all-worlds algorithm for MAB ; however , this would result with a direct dependence on the number of arms # . The challenge is then to improve this dependence by exploiting the structure of the feedback graph . 1.1 Our contributions . We make a significant step towards obtaining a best-of-all-worlds guarantee in the feedback graphs model , by giving the first algorithm in this setting that achieves a uniform best-of-all-worlds guarantee and improves over the existing MAB bounds in a non-trivial way . Our algorithm achieves regret roughly of the form θ ( ) polylog ( ) ) when the losses are stochastic i.i.d. , and Õ ( √ θ ( ) ) ) when the losses are fully adversarial ; here θ ( ) denotes the clique covering number of the graph .1 Furthermore , in an intermediate adversarially-corrupted stochastic setting , where the losses are generated stochastically but then may be corrupted by an adversary , we obtain a bound roughly of the form θ ( ) polylog ( ) ) + Õ ( √ θ ( ) ) with being the total amount of corruption , that smoothly interpolates between these two extremes . Crucially , the algorithm achieves the aforementioned bounds simultaneously and agnostically , in the sense that it need not be aware of the actual underlying loss generation process , and in particular , of the actual level of adversarial corruption introduced . To state our results more concretely , let+1 , . . . , +θ be a minimum clique cover of ( here θ = θ ( ) ) , and denote by ∆ : the minimal gap of a suboptimal arm in + : ; namely , the difference in mean losses ( with respect to the stochastic uncorrupted losses ) between an arm in+ : and the best armwhosemean loss is minimal . Finally , define / = ∑ : :∆ : > 0 1/∆ : ( this is the quantity governing the complexity of the learning problem in the purely stochastic case ; see [ 10 ] ) . Then , our main result can be stated as : Theorem ( main , informal ) . There exists an algorithm ( see Algorithm 1 in Section 4 ) which , given a clique covering +1 , . . . , +θ of , obtains an expected regret bound of Õ ( min { √ θ ) , / + √ / } ) in the -corrupted stochastic setting . In particular , in the stochastic setting ( = 0 ) the expected regret of the algorithm is at most Õ ( ∑ : :∆ : > 0 1/∆ : ) , while in the adversarial setting ( = ) ) its expected regret is bounded by Õ ( √ θ ) ) . The above results leverage the graph structure in a non-trivial way and improve the direct dependence on the number of arms # in existing best-of-all-worlds results to a dependence on θ ( ) , which is potentially significantly smaller than # . On the other hand , our bounds do not feature the optimal dependence on the independence number α ( ) ; we leave it as a main open question whether it is possible to obtain a best-of-all-worlds result with α ( ) replacing θ ( ) in the bounds . We remark however , as we discuss in more detail below , that obtaining bounds scaling with the weaker θ ( ) is already a highly non-trivial task and so we believe that our results form a significant step towards obtaining optimal best-of-all-worlds bounds . Another question we leave open is whether our dependence on ) can be improved toO ( log ) ) andO ( √ ) ) in the stochastic and adversarial cases respectively , removing a few excess logarithmic factors in our bounds . We note that the algorithm achieving the bound above is not adaptive to the best clique covering of the feedback graph for which the quantity / above is minimal . In other words , the algorithm requires a specific minimum clique covering as input , and the regret bounds we show take into account the suboptimality gaps with respect to this given covering , which may be worse than other coverings 1The clique covering number of a graph is the minimum number of cliques that cover its set of vertices ; without loss of generality , we can assume that the latter set of cliques is a partition . with respect to these gaps . Improving the algorithm such that it becomes adaptive to the minimum clique covering could be an interesting direction for future research . 1.2 Overview of main ideas and techniques . Our approach is inspired by recent progress in obtaining best-of-all-worlds guarantees in the two extreme cases of our problem : multi-armed bandits [ 27 ] and full-information experts [ 3 ] . Somewhat surprisingly , it has been shown in both cases that this type of guarantee can be obtained by simple algorithms based on the canonical Follow-the-Regularized-Leader template , traditionally used exclusively in adversarial online learning . Our primary technical challenge therefore boils down to designing a convex regularizer given the feedback graph , that carefully interpolates between the well-understood regularization schemes in the full-information and bandit cases . In the full-information case , best-of-all-worlds results [ 3 ] rely on ( a time-varying version of ) the negative Shannon entropy ' ( ? ) = ∑ # 8=1 ? 8 log ? 8 as regularization . In a nutshell , its crucial properties are an O ( log # ) uniform upper bound on its magnitude over the probability simplex , and on the other hand , that its Hessian ∇2 ' ( ? ) = diag ( ? −11 , . . . , ? −1 # ) satisfies ` T ( ∇2 ' ( ? ) ) −1 ` = O ( 1 ) for any loss vector ` ∈ [ 0 , 1 ] # . ( We omit more technical details on third-order conditions and selfbounding properties from this informal discussion . ) In the bandit case , on the other hand , the best-of-all-worlds result of [ 27 ] relies on a Tsallis entropy ( with parameter α = 1/2 ) regularizer ' ( ? ) = − ∑ # 8=1 √ ? 8 , originally proposed in the context of MAB by [ 4 ] . This form of entropy is bounded uniformly over the simplex byO ( √ # ) , and its Hessian ∇2 ' ( ? ) = 14 diag ( ? −3/2 1 , . . . , ? −3/2 # ) satisfies E [ ˜̀T ( ∇2 ' ( ? ) ) −1 ˜̀ ] = O ( √ # ) for loss estimators ˜̀ used in MAB . In both cases , the regularizer strikes the “ right ” balance between the two bounds , which is necessary for optimal regret . Moving to the more complex feedback model induced by a graph , it is natural to expect that an appropriate regularization would emerge as a certain combination of the Shannon and Tsallis entropies . For simplicity , consider a simple graph formed as a union of θ disjoint cliques+1 , . . . , +θ , and let ? be a probability vector over its vertices . Then , we are looking for a regularization that would treat the marginal clique probabilities @ : = ∑ 8∈+ : ? 8 as a Tsallis entropy , while behaving within a clique ( with respect to the internal conditional probabilities ? 8/ @ : for 8 ∈ + : ) like the negative Shannon entropy . Inspecting the technical conditions more deeply , it turns out that we seek a regularizer ' such that the Hessian at ? is lower bounded by a diagonal matrix with the 8 ’ th diagonal entry corresponding to 8 ∈ + : is ? −18 @ −1/2 : . A natural candidate would be simply the sum of the Shannon and Tsallis entropies ; unfortunately , this regularization lacks the bi-level structure we need , and indeed , its Hessian does not admit the desired lower bound . Our main technical innovation is the design of a new convex regularizer that admits the aforementioned lower bound on its Hessian , and on the other hand , is bounded by O ( √ θ log # ) over the simplex . Interestingly , we find that the necessary conditions are met for a regularizer produced by the product of a negative Tsallis entropy ( over the marginals @ : ) and a negative Shannon entropy ( over the conditional ? 8/ @ : ) , rather than their sum . Of course , such a product need not be a convex function in general , and indeed , the regularization we just described fails to be convex even in simple cases ( see Fig . 1 ) . Nevertheless , it turns out that there is a simple fix that makes this bi-level regularization a convex function : a simple linear shift to the Shannon entropy component . Not only that , but the resulting regularizer also admits the stronger Hessian lower bound we require . We refer to this new regularizer by Tsallis-Shannon entropy , and discuss the details of its derivation in Section 3 ; the remainder of the development takes more standard lines and is described in Sections 4 and 5 .

Summary: This paper studies a best-of-all-worlds algorithm for online learning with feedback graphs by incorporating a novel regularizer. The proposed algorithm simultaneously achieves near-optimal regret w.r.t. $T$ in the adversarial setting, stochastic setting and corrupted stochastic setting.