Minimax Optimal Quantile and Semi-Adversarial Regret via Root-Logarithmic Regularizers

1 Introduction . We focus on the setting of learning with expert advice [ V90 ; LW94 ] , where in each round the learner selects a probability distribution over experts , observes the loss of each expert , and incurs the average loss of the experts under the learner ’ s selected distribution . The learner ’ s objective is to minimize regret against some mixture of the experts , which is the difference between their cumulative loss and the cumulative loss of the expert mixture over T rounds . The classical “ worst-case ” online learning paradigm assumes that the losses are adversarial—that is , they are chosen to make the learner perform as poorly as possible—and demands that the learner competes against the best-performing expert . However , there are many real-world settings where this assumption is too pessimistic , and consequently we focus on designing algorithms with provable guarantees that adapt to easier notions of both data and performance measures . A non-exhaustive list ∗Equal contribution authors . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of work on “ easy data ” includes Freund and Schapire [ FS97 ] , Cesa-Bianchi , Mansour , and Stoltz [ CMS07 ] , van Erven , Grünwald , Koolen , and de Rooij [ v+11 ] , and Gaillard , Stoltz , and van Erven [ GSv14 ] , all of which use variants of the Hedge algorithm [ FS97 ] to obtain regret bounds in terms of data-dependent quantities . Ideally , such quantities are small when the data are “ easy ” to predict . In this work , we focus on two paradigms beyond the classical worst-case : first , we consider relaxing the performance measure to quantile ( KL ) regret , which measures the ability of an algorithm to compete against an unknown mixture of the experts that potentially performs worse than a point-mass on the single best expert , and second , we consider regret within the semi-adversarial paradigm , which defines a spectrum of constraints on the permissible data distributions between stochastic and adversarial . The concept of -quantile regret was introduced by Chaudhuri , Freund , and Hsu [ CFH09 ] , in which the player competes against the b Nc best experts ( out of N total ) rather than the single best . The authors demonstrated empirically that Hedge does poorly in this paradigm , and introduced a new algorithm NormalHedge with an upper bound on quantile regret of √ ( T + ( logN ) 2 ) ( 1 + log ( 1/ ) ) . Later algorithms improved it to √ T ( 1 + log ( 1/ ) ) [ CV10 ; OP16 ] , removing the dependency on N . The semi-adversarial paradigm considers constraining the adversary ’ s choice of data distributions , which was first motivated by Rakhlin , Sridharan , and Tewari [ RST11 ] . Bilodeau , Negrea , and Roy [ BNR20 ] extended this idea , defining adaptive minimax regret with respect to such constraints and providing an efficient algorithm with corresponding regret bounds . Contributions While the best known algorithms for the above two paradigms are intrinsically different , we show that the follow-the-regularized-leader ( FTRL ) algorithm with new root-logarithmic regularizers achieves minimax optimal performance for both quantile and semi-adversarial regret . First , we provide the first FTRL algorithm with minimax optimal quantile regret guarantees , and do so without using the additive normalization step of previous algorithms [ CL06 , Section 2.1 ] . We achieve root-KL bounds that hold uniformly over target distributions on ( possibly uncountable ) expert classes with arbitrary priors , and reduce to the optimal quantile regret for discrete uniform priors . Moreover , we prove matching lower bounds for quantile regret with finite expert classes , demonstrating the optimality of known upper bounds ( including our own ) for the first time . Finally , in the semi-adversarial paradigm , we improve the dependence on the number of experts in the regret bound , obtaining uniformly improved upper bounds over previous work . We achieve the above results through a novel local-norm analysis of FTRL with linearly decomposable regularizers on general ( possibly uncountable ) expert spaces . We use this analysis in conjunction with basic conditions on the first and second derivatives to design and analyze the root-logarithmic regularizers . We believe that this approach is fundamentally different from existing ones and could lead to further advances in obtaining optimal algorithms . In fact , there exist results stating if a regret bound is achievable by some algorithm , then that same bound is nearly achievable by mirror descent with some potential function ( see , e.g. , Srebro , Sridharan , and Tewari [ SST11 , Thm 9 ] ) . However , it is not clear how to design such a function in practice . In contrast , our general FTRL bound reduces the choice of regularizer to a single univariate function , and clarifies how fundamental properties of this function lead to trade-offs in the regret bound . Related Work Chernov and Vovk [ CV10 ] first discussed the fact that the -quantile regret corresponds to the KL divergence between a uniform prior and the uniform mixture of the top b Nc competitors , and such KL bounds have consequently quickly followed quantile regret bounds . Luo and Schapire [ LS14 ; LS15 ] provided a variant of NormalHedge along with a data-dependent KL regret bound at the cost of an additional log T factor ; a similar but even tighter result was obtained by Koolen and van Erven [ Kv15 ] . Orabona and Pál [ OP16 ] showed that these algorithms can be obtained with a reduction to optimal coin-betting online algorithms , yet none of them can be reduced to an FTRL algorithm . Independently , Harvey , Liaw , Perkins , and Randhawa [ H+20 ] proved that a similar strategy allows one to achieve the optimal anytime regret for the setting with N = 2 experts . The first alternative to additively normalized algorithms in the ( bandit ) learning with experts setting was the INF algorithm [ AB09 ] , which was later recast as online mirror descent [ ABL11 ] . Alquier [ A21 ] also did not use additive normalization , and first introduced FTRL with f -divergences ( focusing on the χ2-divergence ) along with regret bounds for continuous distributions with certain unbounded loss functions . For the specific case of Hedge ( and its online mirror descent analogue ) , an analysis on continuous spaces was given by Krichene , Balandat , Tomlin , and Bayen [ K+15 ] and then more generally by van der Hoeven , van Erven , and Kotłowski [ vvK18 ] , while a coarser analysis for f - divergences was given by Alquier [ A21 ] . Finally , choosing regularizers that are tuned to minimize regret for specific tasks has recently led to advances in the online learning literature , both with bandit feedback [ A+17 ; WL18 ; F+16 ] and in the full-information setting [ LWZ18 ] . The best known asymptotic lower bound for learning with expert advice is by Cesa-Bianchi , Freund , Haussler , Helmbold , Schapire , and Warmuth [ C+97 ] , while a finite-time lower bound that asymptotically matches the leading constant was proved by Orabona and Pál [ OP15a ] . We are not aware of lower bounds for the quantile regret , with the notable exception of Koolen [ K13 ] , who proved a lower bound on the regret of learning with two experts based on the KL divergence against some prior . 2 Notation . To analyze FTRL beyond the finite setting requires some more care , and we rely on the language of measure theory to handle finite and uncountable expert classes simultaneously . Let ( Θ , Σ ) be a measurable space , M∞ ( Θ , Σ ) and M ( Θ , Σ ) denote the collection of finite and probability measures respectively , and ν ∈ M∞ ( Θ , Σ ) be arbitrary . A measure q ∈ M∞ ( Θ , Σ ) is absolutely continuous with respect to ν ( denoted q ν ) if q ( A ) = 0 for all A ∈ Σ such that ν ( A ) = 0 . LetMν ( Θ , Σ ) = { q ∈ M ( Θ , Σ ) : q ν } be the set of probability measures that are absolutely continuous with respect to ν . For any integer N , let [ N ] = { 1 , . . . , N } , and simp ( [ N ] ) = { p ∈ [ 0 , 1 ] N : ∑N i=1 pi = 1 } . Let R+ = [ 0 , ∞ ) and R++ = ( 0 , ∞ ) , and define M ≡M ( ( Θ , Σ ) , ( R , R ) ) to be the space of measurable functions from ( Θ , Σ ) to ( R , R ) , whereR is the Borel σ-field on R. Define sets of bounded measurable functions L∞ = { ` ∈M : sup θ∈Θ | ` ( θ ) | < ∞ } and L∞ [ 0,1 ] = { ` ∈M : 0 ≤ ` ≤ 1 } , sets of integrable functions L1 ( ν ) = { x ∈M : ∫ |x ( θ ) | ν ( dθ ) < ∞ } and L1+ ( ν ) = { x ∈ L1 ( ν ) : x ≥ 0 } , and the set of Radon–Nikodym derivatives ( w.r.t . ν ) of probability measures X ( ν ) = { x ∈ L1+ ( ν ) : ∫ x ( θ ) ν ( dθ ) = 1 } . For every x ∈ X ( ν ) , let ν ( x ) ∈ Mν ( Θ , Σ ) denote the unique probability measure satisfying dν ( x ) /dν = x . For every x , z ∈ L∞ , let Conv ( x , z ) = { αx + ( 1− α ) z : α ∈ [ 0 , 1 ] } denote their convex hull . For every ` ∈ L∞ and x ∈ L1 ( ν ) , define 〈 ` , x〉ν = ∫ ` ( θ ) x ( θ ) ν ( dθ ) . Prediction with expert advice We consider the following setting of online linear optimization . For each round t ∈ [ T ] , the player selects µt ∈ M ( Θ , Σ ) based only upon information available prior to round t , and then observes ` t ∈ L∞ . Performance is measured using the regret against some probability measure q ∈M ( Θ , Σ ) , which is defined by RT ( q ) = T∑ t=1 Eθ∼µt [ ` t ( θ ) ] − T∑ t=1 Eθ∼q [ ` t ( θ ) ] . The player ’ s goal is to select µt so that the cumulative loss is not much larger than that of the average loss of an expert under q , and consequently to have small regret . The elements of Θ can be regarded as experts for some prediction problem , and the learner aggregates the predictions of the experts by selecting an expert at random according to µt at round t. For concreteness , note that the usual prediction with expert advice setting for N experts corresponds to M ( Θ , Σ ) = simp ( [ N ] ) . Note that , for any convex loss , applying online linear optimization to the gradients of the losses provides an upper bound on the performance of online convex optimization . Follow-the-regularized-leader Let R = R ∪ { +∞ } . Follow-the-regularized-leader ( FTRL ) [ S07 ; AHR08 ; HK10 ] forms a broad class of algorithms for online convex optimization . For a finite measure ν ∈M∞ ( Θ , Σ ) and a sequence of regularizers ( ψt : X ( ν ) → R ) t∈ [ T ] , ( ψt ) t∈ [ T ] -regularized FTRL is defined by selecting µt+1 = ν ( xt+1 ) using xt+1 ∈ arg min x∈X ( ν ) { 〈Lt , x〉ν + ψt+1 ( x ) } , ( 1 ) where L0 = 0 and Lt = ∑t s=1 ` s. As mentioned above , a classical algorithm for prediction with expert advice on a finite class of N experts is Hedge . While originally analyzed using potential functions , Hedge also corresponds to FTRL where ν is the counting measure and ψt ( · ) = −η−1t H ( · ) for the Shannon entropy H ( x ) =∑ i∈ [ N ] x ( i ) log ( 1/x ( i ) ) with any sequence of regularizer scalings ( ηt ) t∈N ⊆ R++ . Depending on whether the space is continuous or discrete , the range of elements of X ( ν ) will change , and consequently also the minimal domain on which regularizers must be defined . Let ν = inf { ν ( A ) : A ∈ Σ , ν ( A ) > 0 } , and let S ( ν ) = [ 0 , 1/ν ] if ν > 0 and R+ otherwise . By definition , ν ( { θ ∈ Θ : x ( θ ) ∈ S ( ν ) } ) = 1 for all x ∈ X ( ν ) . Concretely , when ν is counting measure then S ( ν ) = [ 0 , 1 ] , when ν is uniform on [ N ] then S ( ν ) = [ 0 , N ] , and when ν is a continuous distribution then S ( ν ) = R+ . In this work , we consider linearly decomposable regularizers of the form ψt ( x ) = η−1t Ψ ν f ( x ) , where Ψνf ( x ) = ∫ f ( x ( θ ) ) ν ( dθ ) ( 2 ) for some f : S ( ν ) → R. We refer to the algorithm that selects µt+1 using Eq . ( 1 ) with a regularizer of the form in Eq . ( 2 ) as Ψνf -regularized FTRL with regularizer scaling ηt .

Summary: The paper studies Follow the Regularised Leader algorithms for the Hedge settings.  The paper starts with an overview of FTRL in adversarial settings. A frighteningly general setup is described and Theorem 1 for the adversarial regret is obtained. It is claimed that the theorem allows one to obtain bounds similar to some known bounds from the literature.  Then the semi-adversarial settings are discussed (the definition of semi-adversarial covers the adversarial settings on one end of the spectrum and purely stochastic on another).  The paper presents FTRL-CARL and obtains a performance bound on it in terms of effective stochastic gaps.  Then the paper discusses bounds in terms of f-divergence and obtains a lower bound matching upper bounds.