Dual Adaptivity: A Universal Algorithm for Minimizing the Adaptive Regret of Convex Functions

To deal with changing environments , a new performance measure—adaptive regret , defined as the maximum static regret over any interval , was proposed in online learning . Under the setting of online convex optimization , several algorithms have been successfully developed to minimize the adaptive regret . However , existing algorithms lack universality in the sense that they can only handle one type of convex functions and need apriori knowledge of parameters . By contrast , there exist universal algorithms , such as MetaGrad , that attain optimal static regret for multiple types of convex functions simultaneously . Along this line of research , this paper presents the first universal algorithm for minimizing the adaptive regret of convex functions . Specifically , we borrow the idea of maintaining multiple learning rates in MetaGrad to handle the uncertainty of functions , and utilize the technique of sleeping experts to capture changing environments . In this way , our algorithm automatically adapts to the property of functions ( convex , exponentially concave , or strongly convex ) , as well as the nature of environments ( stationary or changing ) . As a by product , it also allows the type of functions to switch between rounds . 1 Introduction . Online learning aims to make a sequence of accurate decisions given knowledge of answers to previous tasks and possibly additional information [ Shalev-Shwartz , 2011 ] . It is performed in a sequence of consecutive rounds , where at round t the learner is asked to select a decision wt from a domain Ω . After submitting the answer , a loss function ft : Ω 7→ R is revealed and the learner suffers a loss ft ( wt ) . The standard performance measure is the regret [ Cesa-Bianchi and Lugosi , 2006 ] : Regret ( T ) = T∑ t=1 ft ( wt ) − min w∈Ω T∑ t=1 ft ( w ) defined as the difference between the cumulative loss of the online learner and that of the best decision chosen in hindsight . When both the domain Ω and the loss ft ( · ) are convex , it becomes online convex optimization ( OCO ) [ Zinkevich , 2003 ] . In the literature , there exist plenty of algorithms to minimize the regret under the setting of OCO [ Hazan , 2016 ] . However , when the environment undergoes many changes , regret may not be the best measure of performance . That is because regret chooses a fixed comparator , and for the same reason , it is also referred to as static regret . To address this limitation , Hazan and Seshadhri [ 2007 ] introduce the concept of adaptive regret , which measures the performance with respect to a changing comparator . Following the terminology of Daniely et al . [ 2015 ] , we define the strongly adaptive 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . regret as the maximum static regret over intervals of length τ , i.e. , SA-Regret ( T , τ ) = max [ p , p+τ−1 ] ⊆ [ T ] ( p+τ−1∑ t=p ft ( wt ) − min w∈Ω p+τ−1∑ t=p ft ( w ) ) . ( 1 ) Since the seminal work of Hazan and Seshadhri [ 2007 ] , several algorithms have been successfully developed to minimize the adaptive regret of convex functions , including general convex , exponentially concave ( abbr . exp-concave ) and strongly convex functions [ Hazan and Seshadhri , 2009 , Jun et al. , 2017a , Zhang et al. , 2018b ] . However , existing methods can only handle one type of convex functions . Furthermore , when facing exp-concave and strongly convex functions , they need to know the moduli of exp-concavity and strong convexity . The lack of universality hinders their applications to real-world problems . On the other hand , there do exist universal algorithms , such as MetaGrad [ van Erven and Koolen , 2016 ] , that attain optimal static regret for multiple types of convex functions simultaneously . This observation motivates us to ask whether it is possible to design a single algorithm to minimize the adaptive regret of multiple types of functions . This is very challenging because the algorithm needs to enjoy dual adaptivity , adaptive to the function type and adaptive to the environment . In this paper , we provide an affirmative answer by developing a Universal algorithm for Minimizing the Adaptive regret ( UMA ) . First , inspired by MetaGrad , UMA maintains multiple learning rates to handle the uncertainty of functions . In this way , it supports multiple types of functions simultaneously and identifies the best learning rate automatically . Second , following existing studies on adaptive regret , UMA deploys sleeping experts [ Freund et al. , 1997 ] to minimize the regret over any interval , and thus achieves a small adaptive regret and captures the changing environment . The main advantage of UMA is that it attains second-order regret bounds over any interval . As a result , it can minimize the adaptive regret of general convex functions , and automatically take advantage of easier functions whenever possible . Specifically , UMA attains O ( √ τ log T ) , O ( dα log τ log T ) and O ( 1λ log τ log T ) strongly adaptive regrets for general convex , α-exp-concave and λ-strongly convex functions respectively , where d is the dimensionality . All of these bounds match the state-of-the-art results on adaptive regret [ Jun et al. , 2017a , Zhang et al. , 2018b ] exactly . Furthermore , UMA can also handle the case that the type of functions changes between rounds . For example , suppose the online functions are general convex during interval I1 , then become α-exp-concave in I2 , and finally switch to λ-strongly convex in I3 . When facing this function sequence , UMA achieves O ( √ |I1| log T ) , O ( dα log |I2| log T ) and O ( 1 λ log |I3| log T ) regrets over intervals I1 , I2 and I3 , respectively . 2 Related work . We briefly review related work on static regret and adaptive regret , under the setting of OCO . 2.1 Static regret . To minimize the static regret of general convex functions , online gradient descent ( OGD ) with step size ηt = O ( 1/ √ t ) achieves an O ( √ T ) bound [ Zinkevich , 2003 ] . If all the online functions are λ-strongly convex , OGD with step size ηt = O ( 1/ [ λt ] ) attains anO ( 1λ log T ) bound [ Shalev-Shwartz et al. , 2007 ] . When the functions are α-exp-concave , online Newton step ( ONS ) , with knowledge of α , enjoys an O ( dα log T ) bound , where d is the dimensionality [ Hazan et al. , 2007 ] . These regret bounds are minimax optimal for the corresponding type of functions [ Abernethy et al. , 2008 ] , but choosing the optimal algorithm for a specific problem requires domain knowledge . The study of universal algorithms for OCO stems from the adaptive online gradient descent ( AOGD ) [ Bartlett et al. , 2008 ] and its proximal extension [ Do et al. , 2009 ] . The key idea of AOGD is to add a quadratic regularization term to the loss . Bartlett et al . [ 2008 ] demonstrate that AOGD is able to interpolate between the O ( √ T ) bound of general convex functions and the O ( log T ) bound of strongly convex functions . Furthermore , it allows the online function to switch between general convex and strongly convex . However , AOGD has two restrictions : • It needs to calculate the modulus of strong convexity on the fly , which is a nontrivial task ; • It does not support exp-concave functions explicitly , and thus can only achieve a suboptimal O ( √ T ) regret for this type of functions . Another milestone is the multiple eta gradient algorithm ( MetaGrad ) [ van Erven and Koolen , 2016 , Mhammedi et al. , 2019 , van Erven et al. , 2021 ] , which adapts to a much broader class of functions , including convex and exp-concave functions . MetaGrad ’ s main feature is that it simultaneously considers multiple learning rates and does not need to know the modulus of exp-concavity . MetaGrad achieves O ( √ T log log T ) and O ( dα log T ) regret bounds for general convex and α-exp-concave functions , respectively . However , it suffers the following two limitations : • MetaGrad treats strongly convex functions as exp-concave , and thus only gives a suboptimal O ( dλ log T ) regret for λ-strongly convex functions ; • It assumes the type of online functions , as well as the associated parameter , does not change between rounds . The first limitation of MetaGrad has been addressed by Wang et al . [ 2019 ] , who develop a universal algorithm named as multiple sub-algorithms and learning rates ( Maler ) . It attainsO ( √ T ) , O ( dα log T ) and O ( 1λ log T ) regret bounds for general convex , α-exp-concave , and λ-strongly convex functions , respectively . Furthermore , Wang et al . [ 2020 ] extend Maler to make use of smoothness . However , the second limitation remains there . 2.2 Adaptive regret . Adaptive regret has been studied in the setting of prediction with expert advice [ Littlestone and Warmuth , 1994 , Freund et al. , 1997 , Adamskiy et al. , 2012 , György et al. , 2012 , Luo and Schapire , 2015 ] and OCO [ Hazan and Seshadhri , 2007 , Daniely et al. , 2015 , Jun et al. , 2017a ] . In this section , we focus on the related work in the latter one . Adaptive regret is introduced by Hazan and Seshadhri [ 2007 ] , and later refined by Daniely et al . [ 2015 ] . To distinguish between them , we refer to the definition of Hazan and Seshadhri as weakly adaptive regret : WA-Regret ( T ) = max [ p , q ] ⊆ [ T ] ( q∑ t=p ft ( wt ) − min w∈Ω q∑ t=p ft ( w ) ) . For α-exp-concave functions , Hazan and Seshadhri [ 2007 ] propose an adaptive algorithm named as Follow-the-Leading-History ( FLH ) . FLH restarts a copy of ONS in each round as an expert , and chooses the best one using expert-tracking algorithms . The meta-algorithm used to track the best expert is inspired by the Fixed-Share algorithm [ Herbster and Warmuth , 1998 ] . While FLH is equipped with an O ( dα log T ) weakly adaptive regret , it is computationally expensive since it needs to maintain t experts in the t-th iteration . To reduce the computational cost , Hazan and Seshadhri [ 2007 ] further prune the number of experts based on a data streaming algorithm . In this way , FLH only keeps O ( log t ) experts , at the price of an O ( dα log 2 T ) weakly adaptive regret . Notice that the efficient version of FLH essentially creates and removes experts dynamically . As pointed out by Adamskiy et al . [ 2012 ] , this behavior can be modeled by the sleeping expert setting [ Freund et al. , 1997 ] , in which the expert can be “ asleep ” for certain rounds and does not make any advice . For general convex functions , we can use OGD as the expert-algorithm in FLH . Hazan and Seshadhri [ 2007 ] prove that FLH and its efficient variant attain O ( √ T log T ) and O ( √ T log3 T ) weakly adaptive regrets , respectively . This result reveals a limitation of weakly adaptive regret—it does not respect short intervals well . For example , the O ( √ T log T ) regret bound is meaningless for intervals of length O ( √ T ) . To address this limitation , Daniely et al . [ 2015 ] introduce the strongly adaptive regret which takes the interval length as a parameter , as shown in ( 1 ) . They propose a novel meta-algorithm , named as Strongly Adaptive Online Learner ( SAOL ) . SAOL carefully constructs a set of intervals , then runs an instance of low-regret algorithm in each interval as an expert , and finally combines active experts ’ outputs by a variant of multiplicative weights method [ Arora et al. , 2012 ] . SAOL also maintains O ( log t ) experts in the t-th round , and achieves an O ( √ τ log T ) strongly adaptive regret for convex functions . Later , Jun et al . [ 2017a ] develop a new meta-algorithm named as sleeping coin betting ( SCB ) , and improve the strongly adaptive regret bound to O ( √ τ log T ) . Very recently , Cutkosky [ 2020 ] proposes a novel strongly adaptive method , which can guarantee the O ( √ τ log T ) bound in the worst case , while achieving tighter results when the square norms of gradients are small . For λ-strongly convex functions , Zhang et al . [ 2018b ] point out that we can replace ONS with OGD , and obtain an O ( 1λ log T ) weakly adaptive regret . They also demonstrate that the number of active experts can be reduced from t to O ( log t ) , at a cost of an additional log T factor in the regret . All the aforementioned adaptive algorithms need to query the gradient of the loss function at least Θ ( log t ) times in the t-th iteration . Based on surrogate losses , Wang et al . [ 2018 ] show that the number of gradient evaluations per round can be reduced to 1 without affecting the performance . When we have prior knowledge about the change of environments , it is also possible to improve the logarithmic factor in the adaptive regret [ Wan et al. , 2021 ] .

Summary: In this papers, the authors tackle the problem of online optimization with unknown (or even changing) convex function types, which can be plain convex, exp-concave, or strongly convex. Their main goal is to design an algorithm that suffers asymptotically optimal regret on any time interval.   To this end they extend the MetaGrad algorithm, a method that guarantees low static regret against plain convex or exp-concave functions, to also handle strongly convex functions and in a way that guarantees low adaptive regret, not only static regret. Their main technical contribution is showing that this can be done by incorporating sleeping experts in the meta algorithm of MetaGrad.