Minimax Optimal Quantile and Semi-Adversarial Regret via Root-Logarithmic Regularizers

1 Introduction . We focus on the setting of learning with expert advice [ V90 ; LW94 ] , where in each round the learner selects a probability distribution over experts , observes the loss of each expert , and incurs the average loss of the experts under the learner ’ s selected distribution . The learner ’ s objective is to minimize regret against some mixture of the experts , which is the difference between their cumulative loss and the cumulative loss of the expert mixture over T rounds . The classical “ worst-case ” online learning paradigm assumes that the losses are adversarial—that is , they are chosen to make the learner perform as poorly as possible—and demands that the learner competes against the best-performing expert . However , there are many real-world settings where this assumption is too pessimistic , and consequently we focus on designing algorithms with provable guarantees that adapt to easier notions of both data and performance measures . A non-exhaustive list ∗Equal contribution authors . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of work on “ easy data ” includes Freund and Schapire [ FS97 ] , Cesa-Bianchi , Mansour , and Stoltz [ CMS07 ] , van Erven , Grünwald , Koolen , and de Rooij [ v+11 ] , and Gaillard , Stoltz , and van Erven [ GSv14 ] , all of which use variants of the Hedge algorithm [ FS97 ] to obtain regret bounds in terms of data-dependent quantities . Ideally , such quantities are small when the data are “ easy ” to predict . In this work , we focus on two paradigms beyond the classical worst-case : first , we consider relaxing the performance measure to quantile ( KL ) regret , which measures the ability of an algorithm to compete against an unknown mixture of the experts that potentially performs worse than a point-mass on the single best expert , and second , we consider regret within the semi-adversarial paradigm , which defines a spectrum of constraints on the permissible data distributions between stochastic and adversarial . The concept of -quantile regret was introduced by Chaudhuri , Freund , and Hsu [ CFH09 ] , in which the player competes against the b Nc best experts ( out of N total ) rather than the single best . The authors demonstrated empirically that Hedge does poorly in this paradigm , and introduced a new algorithm NormalHedge with an upper bound on quantile regret of √ ( T + ( logN ) 2 ) ( 1 + log ( 1/ ) ) . Later algorithms improved it to √ T ( 1 + log ( 1/ ) ) [ CV10 ; OP16 ] , removing the dependency on N . The semi-adversarial paradigm considers constraining the adversary ’ s choice of data distributions , which was first motivated by Rakhlin , Sridharan , and Tewari [ RST11 ] . Bilodeau , Negrea , and Roy [ BNR20 ] extended this idea , defining adaptive minimax regret with respect to such constraints and providing an efficient algorithm with corresponding regret bounds . Contributions While the best known algorithms for the above two paradigms are intrinsically different , we show that the follow-the-regularized-leader ( FTRL ) algorithm with new root-logarithmic regularizers achieves minimax optimal performance for both quantile and semi-adversarial regret . First , we provide the first FTRL algorithm with minimax optimal quantile regret guarantees , and do so without using the additive normalization step of previous algorithms [ CL06 , Section 2.1 ] . We achieve root-KL bounds that hold uniformly over target distributions on ( possibly uncountable ) expert classes with arbitrary priors , and reduce to the optimal quantile regret for discrete uniform priors . Moreover , we prove matching lower bounds for quantile regret with finite expert classes , demonstrating the optimality of known upper bounds ( including our own ) for the first time . Finally , in the semi-adversarial paradigm , we improve the dependence on the number of experts in the regret bound , obtaining uniformly improved upper bounds over previous work . We achieve the above results through a novel local-norm analysis of FTRL with linearly decomposable regularizers on general ( possibly uncountable ) expert spaces . We use this analysis in conjunction with basic conditions on the first and second derivatives to design and analyze the root-logarithmic regularizers . We believe that this approach is fundamentally different from existing ones and could lead to further advances in obtaining optimal algorithms . In fact , there exist results stating if a regret bound is achievable by some algorithm , then that same bound is nearly achievable by mirror descent with some potential function ( see , e.g. , Srebro , Sridharan , and Tewari [ SST11 , Thm 9 ] ) . However , it is not clear how to design such a function in practice . In contrast , our general FTRL bound reduces the choice of regularizer to a single univariate function , and clarifies how fundamental properties of this function lead to trade-offs in the regret bound . Related Work Chernov and Vovk [ CV10 ] first discussed the fact that the -quantile regret corresponds to the KL divergence between a uniform prior and the uniform mixture of the top b Nc competitors , and such KL bounds have consequently quickly followed quantile regret bounds . Luo and Schapire [ LS14 ; LS15 ] provided a variant of NormalHedge along with a data-dependent KL regret bound at the cost of an additional log T factor ; a similar but even tighter result was obtained by Koolen and van Erven [ Kv15 ] . Orabona and Pál [ OP16 ] showed that these algorithms can be obtained with a reduction to optimal coin-betting online algorithms , yet none of them can be reduced to an FTRL algorithm . Independently , Harvey , Liaw , Perkins , and Randhawa [ H+20 ] proved that a similar strategy allows one to achieve the optimal anytime regret for the setting with N = 2 experts . The first alternative to additively normalized algorithms in the ( bandit ) learning with experts setting was the INF algorithm [ AB09 ] , which was later recast as online mirror descent [ ABL11 ] . Alquier [ A21 ] also did not use additive normalization , and first introduced FTRL with f -divergences ( focusing on the χ2-divergence ) along with regret bounds for continuous distributions with certain unbounded loss functions . For the specific case of Hedge ( and its online mirror descent analogue ) , an analysis on continuous spaces was given by Krichene , Balandat , Tomlin , and Bayen [ K+15 ] and then more generally by van der Hoeven , van Erven , and Kotłowski [ vvK18 ] , while a coarser analysis for f - divergences was given by Alquier [ A21 ] . Finally , choosing regularizers that are tuned to minimize regret for specific tasks has recently led to advances in the online learning literature , both with bandit feedback [ A+17 ; WL18 ; F+16 ] and in the full-information setting [ LWZ18 ] . The best known asymptotic lower bound for learning with expert advice is by Cesa-Bianchi , Freund , Haussler , Helmbold , Schapire , and Warmuth [ C+97 ] , while a finite-time lower bound that asymptotically matches the leading constant was proved by Orabona and Pál [ OP15a ] . We are not aware of lower bounds for the quantile regret , with the notable exception of Koolen [ K13 ] , who proved a lower bound on the regret of learning with two experts based on the KL divergence against some prior . 2 Notation . To analyze FTRL beyond the finite setting requires some more care , and we rely on the language of measure theory to handle finite and uncountable expert classes simultaneously . Let ( Θ , Σ ) be a measurable space , M∞ ( Θ , Σ ) and M ( Θ , Σ ) denote the collection of finite and probability measures respectively , and ν ∈ M∞ ( Θ , Σ ) be arbitrary . A measure q ∈ M∞ ( Θ , Σ ) is absolutely continuous with respect to ν ( denoted q ν ) if q ( A ) = 0 for all A ∈ Σ such that ν ( A ) = 0 . LetMν ( Θ , Σ ) = { q ∈ M ( Θ , Σ ) : q ν } be the set of probability measures that are absolutely continuous with respect to ν . For any integer N , let [ N ] = { 1 , . . . , N } , and simp ( [ N ] ) = { p ∈ [ 0 , 1 ] N : ∑N i=1 pi = 1 } . Let R+ = [ 0 , ∞ ) and R++ = ( 0 , ∞ ) , and define M ≡M ( ( Θ , Σ ) , ( R , R ) ) to be the space of measurable functions from ( Θ , Σ ) to ( R , R ) , whereR is the Borel σ-field on R. Define sets of bounded measurable functions L∞ = { ` ∈M : sup θ∈Θ | ` ( θ ) | < ∞ } and L∞ [ 0,1 ] = { ` ∈M : 0 ≤ ` ≤ 1 } , sets of integrable functions L1 ( ν ) = { x ∈M : ∫ |x ( θ ) | ν ( dθ ) < ∞ } and L1+ ( ν ) = { x ∈ L1 ( ν ) : x ≥ 0 } , and the set of Radon–Nikodym derivatives ( w.r.t . ν ) of probability measures X ( ν ) = { x ∈ L1+ ( ν ) : ∫ x ( θ ) ν ( dθ ) = 1 } . For every x ∈ X ( ν ) , let ν ( x ) ∈ Mν ( Θ , Σ ) denote the unique probability measure satisfying dν ( x ) /dν = x . For every x , z ∈ L∞ , let Conv ( x , z ) = { αx + ( 1− α ) z : α ∈ [ 0 , 1 ] } denote their convex hull . For every ` ∈ L∞ and x ∈ L1 ( ν ) , define 〈 ` , x〉ν = ∫ ` ( θ ) x ( θ ) ν ( dθ ) . Prediction with expert advice We consider the following setting of online linear optimization . For each round t ∈ [ T ] , the player selects µt ∈ M ( Θ , Σ ) based only upon information available prior to round t , and then observes ` t ∈ L∞ . Performance is measured using the regret against some probability measure q ∈M ( Θ , Σ ) , which is defined by RT ( q ) = T∑ t=1 Eθ∼µt [ ` t ( θ ) ] − T∑ t=1 Eθ∼q [ ` t ( θ ) ] . The player ’ s goal is to select µt so that the cumulative loss is not much larger than that of the average loss of an expert under q , and consequently to have small regret . The elements of Θ can be regarded as experts for some prediction problem , and the learner aggregates the predictions of the experts by selecting an expert at random according to µt at round t. For concreteness , note that the usual prediction with expert advice setting for N experts corresponds to M ( Θ , Σ ) = simp ( [ N ] ) . Note that , for any convex loss , applying online linear optimization to the gradients of the losses provides an upper bound on the performance of online convex optimization . Follow-the-regularized-leader Let R = R ∪ { +∞ } . Follow-the-regularized-leader ( FTRL ) [ S07 ; AHR08 ; HK10 ] forms a broad class of algorithms for online convex optimization . For a finite measure ν ∈M∞ ( Θ , Σ ) and a sequence of regularizers ( ψt : X ( ν ) → R ) t∈ [ T ] , ( ψt ) t∈ [ T ] -regularized FTRL is defined by selecting µt+1 = ν ( xt+1 ) using xt+1 ∈ arg min x∈X ( ν ) { 〈Lt , x〉ν + ψt+1 ( x ) } , ( 1 ) where L0 = 0 and Lt = ∑t s=1 ` s. As mentioned above , a classical algorithm for prediction with expert advice on a finite class of N experts is Hedge . While originally analyzed using potential functions , Hedge also corresponds to FTRL where ν is the counting measure and ψt ( · ) = −η−1t H ( · ) for the Shannon entropy H ( x ) =∑ i∈ [ N ] x ( i ) log ( 1/x ( i ) ) with any sequence of regularizer scalings ( ηt ) t∈N ⊆ R++ . Depending on whether the space is continuous or discrete , the range of elements of X ( ν ) will change , and consequently also the minimal domain on which regularizers must be defined . Let ν = inf { ν ( A ) : A ∈ Σ , ν ( A ) > 0 } , and let S ( ν ) = [ 0 , 1/ν ] if ν > 0 and R+ otherwise . By definition , ν ( { θ ∈ Θ : x ( θ ) ∈ S ( ν ) } ) = 1 for all x ∈ X ( ν ) . Concretely , when ν is counting measure then S ( ν ) = [ 0 , 1 ] , when ν is uniform on [ N ] then S ( ν ) = [ 0 , N ] , and when ν is a continuous distribution then S ( ν ) = R+ . In this work , we consider linearly decomposable regularizers of the form ψt ( x ) = η−1t Ψ ν f ( x ) , where Ψνf ( x ) = ∫ f ( x ( θ ) ) ν ( dθ ) ( 2 ) for some f : S ( ν ) → R. We refer to the algorithm that selects µt+1 using Eq . ( 1 ) with a regularizer of the form in Eq . ( 2 ) as Ψνf -regularized FTRL with regularizer scaling ηt .

Summary: This paper studies the classic expert problem and focuses on developing  parameter-free algorithms with two specific guarantees: 1) better regret  bounds for the semi-adversarial paradigm proposed by Bilodeau et al. 20;  2) a KL-type regret bound similar to that achieved by NormalHedge  (Chaudhuri et al. 09) and other variants, possibly for an uncountable  set of experts. The authors achieve these two goals using the standard  FTRL framework with two new root-logarithmic regularizers, inspired by  their general analysis that requires some specific properties of the  regularizer. A matching lower bound for the KL-type regret is also  provided.