Hybrid Regret Bounds for Combinatorial Semi-Bandits and Adversarial Linear Bandits

1 Introduction . In this work , we consider two fundamental problem settings w.r.t . online decision problems : combinatorial semi-bandits [ 42 , 66 , 4 ] and linear bandits [ 13 , 17 , 30 ] . In both problem settings , a player is given an action set A ∈ Rd , a compact subset of a d-dimensional vector space . In each round t , the player chooses an action at ∈ A and then incurs loss ` > t at , where ` t ∈ Rd is a loss vector chosen by the environment . The action set in the combinatorial semi-bandit is assumed to be a subset of { 0 , 1 } d , each element of which corresponds to a subset of [ d ] = { 1 , 2 , . . . , d } . After choosing at ∈ A , the player can observe ` ti for all i such that ati = 1 in semi-bandits . Linear bandits are problems with even more limited feedback , ones in which the learner can only observe the incurred loss ` > t at . For combinatorial semi-bandits problems with ` t ∈ [ 0 , 1 ] d , there is a known algorithm with an O ( √ mdT ) -regret bound [ 4 ] , where m = maxa∈A ‖a‖1 . For linear bandits such that | ` > t a| ≤ 1 holds for any a ∈ A , algorithms with Õ ( d √ T ) -regret ( where factors in log T and log d are ignored ) have been developed [ 13 , 17 , 30 ] . These algorithms are optimal in terms of worst-case analysis . In fact , matching lower bounds of Ω ( √ mdT ) for combinatorial semi-bandits [ 3 , 11 ] and of Ω ( d √ T ) for linear bandits [ 22 ] are known . The worst-case optimal algorithms , however , tend to be too conservative in actual practice . This is because true worst-case environments are quite rare in real-world applications . Rather , the environments may have structures that are convenient for the learner , and it is desirable that the algorithm takes advantage of such structures to improve performance . To exploit such structures , two main categories of approaches have been studied : adapting to ( nearly ) stochastic environments and developing data-dependent regret bounds . An example of the first category is in reference to best-of-both-worlds ( BOBW ) algorithms [ 12 , 62 , 5 , 61 , 68 , 70 ] , which means that they work well for both adversarial and stochastic settings . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . These algorithms enjoy Õ ( √ T ) -regret bounds in an adversarial setting and , simultaneously , achieve O ( ( log T ) c ∆ ) -regret in a stochastic setting with i.i.d . losses , where c ≥ 1 is a constant and ∆ represents the suboptimality gap defined by mina∈A\a∗ E [ ` > t ( a− a∗ ) ] for an optimal action a∗ ∈ A . As shown in Table 1 , for combinatorial semi-bandits , Zimmert et al . [ 71 ] provide a BOBW algorithm , both bounds of which are tight up to constant factors as matching lower bounds are known for adversarial settings [ 3 , 11 ] as well as for stochastic settings [ 42 ] . Studies on the adversarial robustness of stochastic bandit algorithms [ 48 , 27 , 70 ] can be considered to provide another approach in the first category , in which an adversary can corrupt stochasticallygenerated losses subject to the constraint that the total amount of the corruption is at most a parameter C ≥ 0 , referred to as the corruption level . This model includes both adversarial and stochastic settings and is closely related to studies on BOBW algorithms . In fact , the special cases of C = 0 and C = Ω ( T ) correspond to stochastic and adversarial settings , respectively . For the fundamental multiarmed bandit problem , Zimmert and Seldin [ 70 ] have proposed the Tsallis-INF algorithm , which achieves BOBW with tight regret bounds up to small constant factors , and which simultaneously is very robust w.r.t . corruptions ; the degradation in the regret is only O ( min { C , √ CK log T ∆ } ) . Such algorithms are called best-of-three-worlds ( BOTW ) algorithms . As shown in Table 1 , the semi-bandit algorithm by Zimmert et al . [ 71 ] is also a BOTW algorithm ( Sto . +Adv . refers to the stochastic setting with C-adversarial corruptions ) .1 For linear bandits , Lee et al . [ 45 ] have recently developed a BOTW algorithm as well . In studies on data-dependent regret bounds [ 55 , 68 , 29 , 57 , 15 , 14 , 34 ] of the other category , we define measures of the difficulty of problem instances , which we refer to as difficulty indicators , and aim to develop algorithms so that the regret will be smaller for smaller instance-difficulty indicators . Examples of difficulty indicators dealt with in this paper include L∗ : the cumulative losses for an optimal action , Qq : the total quadratic variation of losses , and Vq : the path-length of losses , definitions of which are given in Table 3 . Tables 1 and 2 show data-dependent regret bounds , including difficulty indicators for , respectively , combinatorial semi-bandits and adversarial linear bandits . For example , the semi-bandit algorithm with an O ( √ dL∗ log T ) -regret bound given by Wei and Luo [ 68 ] achieves much smaller regret than the worst-case optimal O ( √ dmT ) -bound when L∗ = o ( mT ) , i.e. , when there exists an action for which the cumulative losses are much smaller than mT . Given the various algorithms above , a new challenge arises : how , in practice , can we choose an appropriate algorithm ? If the environment is expected to behave in an almost i.i.d . stochastic manner , either BOBW or BOTW algorithms would work well . If the environment is far from exhibiting stochastic behavior but is `` easy '' in terms of some difficulty indicator , algorithms with a corresponding data-dependent regret bound may work better . In practice , however , it is hard to tell what kind of environment we are working in until we have actually tried out the algorithm . 1.1 Contributions of this work . This work addresses the above-mentioned issue of algorithm selection by developing hybrid algorithms . Its main contribution is to develop a semi-bandit algorithm ( Algorithm 1 ) that enjoys multiple data-dependent regret bounds as well . This can be seen as an an extension of the work on the multi-armed bandit problem by Ito [ 34 ] to combinatorial semi-bandits . In addition to this , for linear bandits , we provide a hybrid data-dependent regret bound . For combinatorial semi-bandits problems , we propose Algorithm 1 with the regret bounds shown in Table 1 . More explicit statements are provided in Theorem 1 , Corollary 1 ( for the adversarial setting ) , and Corollary 2 ( for the stochastic setting with/without adversarial corruption ) . As can be seen in the table , the regret bound for the adversarial setting encompasses three different data-dependent regret bounds . Note that O ( √ dL∗ log T ) -bounds imply the nearly worst-case optimal bound of O ( √ dmT log T ) , as L∗ ≤ mT always follows from the model definition . The new regret bound is of O ( R+ √ RCm ) for the corrupted stochastic setting , where R stands for the bound for the stochastic setting . Note that the regret bounds for the stochastic settings ( with corruption ) can be improved for some special cases , such as for size-invariant ( ‖a‖1 = m for all a ∈ A ) or for a matroid constraint ( A forms bases of a matroid ) , as described in Corollary 2 . We would also like to stress that the 1This bound is not explicitly stated in their paper , but can be shown via a straightforward modification of the proof . A proof of this is given in Appendix B of the supplementary material . proposed algorithm is parameter-free , i.e. , it does not require any prior information w.r.t . parameters ∆ , L∗ , Q2 , V1 , and C. The proposed semi-bandit algorithm is based on a follow-the-regularized-leader ( FTRL ) framework [ 70 , 71 ] , combined with an optimistic prediction for the losses [ 58 , 57 , 68 ] . More precisely , it uses a mixture regularizer [ 14 , 55 , 71 , 24 ] consisting of the log-barrier in variables xi and the Shannon entropy in the complement ( 1 − xi ) of xi with entry-wisely adaptive learning rates , by which BOTW is achieved . The regret analysis for the stochastic setting ( with corruption ) is based on self-bounding inequalities for the regret , similarly to what is seen in the analyses by Zimmert et al . [ 71 ] , Zimmert and Seldin [ 70 ] . In addition to this , by choosing optimistic predictors with simple gradient descent methods , we can achieve multiple data-dependent regret bounds as well . The most relevant algorithms are given by Wei and Luo [ 68 ] and Zimmert et al . [ 71 ] . The proposed algorithm differs from that of Wei and Luo [ 68 ] in that the former employs follow-the-regularized-leader rather than online mirror descent methods and uses a mixture regularizer . The major differences with the work by Zimmert et al . [ 71 ] are that Algorithm 1 uses a log-barrier rather than Tsallis entropy , and that it takes advantages of optimistic predictors . As far as we have managed to determine , it appears difficult to combine Tsallis-entropy-based algorithms with optimistic predictors , as we discuss in Subsection 4.2 . Our work overcomes this difficulty by using a log-barrier regularizer , rather than Tsallis entropy , in exchange for additional O ( √ log T ) -factors in worst-case regret bounds . Remark 1 . In the previously mentioned study by Wei and Luo [ 68 ] , the regret bounds for combinatorial semi-bandits are not given explicitly . However , as stated just after Corollary 3 in their paper ( `` but they can be straightforwardly generalized to the semi-bandit case '' ) , we can obtain the regret bounds in Table 1 via a simple calculation . To be more precise , their regret bounds in Table 1 dependent on Q2 and V1 can be refined by replacing them with Q2 ( I∗ ) : = ∑T t=1 ∑ i∈I∗ ( ` ti − ¯̀i ) 2 and V2 ( I∗ ) : = ∑T−1 t=1 ∑ i∈I∗ | ` ti − ` t+1 , i| , respectively , where we define I∗ = { i ∈ [ d ] | a∗i = 1 } for an optimal action a∗ . This means that Algorithm 1 is not necessarily superior to their algorithms . On the other hand , it should be noted that their algorithms require prior knowledge w.r.t . Q2 ( I∗ ) and V2 ( I ∗ ) to achieve corresponding regret bounds . For linear bandits problems , we provide the hybrid data-dependent regret bounds shown in Table 2 and in Theorem 3 , which holds for p ∈ [ 2 , ∞ ] and q ∈ [ 1 , 2 ] such that 1/p + 1/q = 1 , under the assumption of ‖a‖p ≤ 1 for all a ∈ A and ‖ ` t‖q ≤ 1 for all t. The parameter ϑ ≥ 1 is associated with a self-concordant barrier over the convex hull of A . It is known that any convex set has a self-concordant barrier with ϑ = O ( d ) [ 53 ] , which is tight up to a constant factor . Substituting ϑ = d and noting L∗ ≤ T , we can see that the new regret bound includes previous bounds . Further , for the special case of ( p , q ) = ( ∞ , 1 ) , the new path-length regret bound of Õ ( d √ V1 ) is tight up to a logarithmic factor in T , as a matching lower bound of O ( d √ V1 ) is known [ 22 , 15 ] . To our knowledge , this is the first ( nearly ) tight path-length regret bound for linear bandits . The regret bounds of Õ ( d √ L∗ ) and Õ ( d √ Qq ) are also nearly tight , as has been noted in the literature [ 36 ] . The approach for the new regret bound is quite simple : we combine regret bounds dependent on optimistic predictors [ 57 , 36 ] and the algorithm of tracking the best linear predictor [ 31 , 16 ] .

Summary: This paper studies the hybrid regret bounds for combinatorial semi-bandits (in stochastic, stochastic with corruption and adversarial settings simultaneously) and linear bandits (w.r.t. three different parameters). The paper improves the previous results by combining optimistic predictors with FTRL and the alg. of tracking the best linear predictor.