Fast rates for prediction with limited expert advice

√ T ) with probability lower bounded by a constant . However , if we are allowed to see at least two actively chosen expert advices per training round and use at least two experts for prediction , the fast rate O ( 1/T ) can be achieved . We design novel algorithms achieving this rate in this setting , and in the setting where the learner has a budget constraint on the total number of observed expert advices , and give precise instance-dependent bounds on the number of training rounds and queries needed to achieve a given generalization error precision . Keywords : Online Learning , Budgeted Learning , Prediction with expert advice . 1 Introduction and setting . We consider a generic prediction problem in a stochastic setting : a target random variable Y taking values in Y is to be predicted by a user-determined forecast F , also modeled as a random variable , taking values in a closed convex subset X of Rd . The mismatch between the two is measured via a loss function l ( F , Y ) . The quality of the agent ’ s output is measured by its generalization risk R ( F ) : = E [ l ( F , Y ) ] . To assist us in this task , the forecast or “ advice ” of a number of “ experts ” ( F1 , . . . , FK ) ( also modeled as random variables ) can be requested . The agent ’ s objective is to achieve a risk as close as possible to the risk of the best expert R∗ = mini∈JKKR ( Fi ) ( for a nonnegative integer n , we denote JnK = { 1 , . . . , n } ) . We measure the performance of the user ’ s forecast via its excess risk ( or average regret ) with respect to that best expert . The literature on expert advice generally considers the cumulative regret over a sequence of forecasts Ft followed by observation of the target variable Yt and incurring the loss l ( Ft , Yt ) , t = 1 , . . . , T . In the present work we will separate observation ( or training ) phase and forecast phase : the user is allowed to observe ( some of ) the expert ’ s predictions and the target variable for a number of independent , identically distributed rounds ( Yt , F1 , t , . . . , FK , t ) 1≤t≤T following certain rules to be specified . After the observation phase , the user must decide of a prediction strategy , namely a convex combination of the experts F̂ = ∑k i=1 ŵiFi , where the weights ŵi can be chosen based on the information gathered in the training phase . The risk of this strategy is R ( F̂ ) , where the risk is evaluated on new , independent data . In other words , if the training phase takes place over T independent rounds , the forecast risk is the expected loss over the ( T + 1 ) th , independent , round . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In some situations , it may be overly expensive to query the advice of all experts at each round . The cost can be monetary if each expert demands to be paid to reveal his opinion , possibly because they have access to some information that others do not . In this case we may have a total limit on how much we can spend . In a different context , it is unrealistic to ask for the advice of all available doctors or to run a large battery of tests on each patient . In this case , we may be have a strong limit on the number of expert opinions that can be consulted for each training instance . In a more typical machine learning scenario , each “ expert ” might be a fixed prediction method Fi = fi ( X ) ( using the information of a covariate X ) , where the predictor functions fi have been already trained in advance , albeit based on different sets of parameters or methodology ; the goal then amounts to predictor selection or aggregation , in a situation where the computation of each single prediction constitutes the bottleneck cost , rather than data acquisition . Overall the agent ’ s goal is to achieve a risk close to optimal while sparing on the number of experts queries – both at training time and for forecast . Motivated by these questions we investigate several scenarios for prediction with limited access to expert advice . Furthermore , our emphasis is on obtaining fast convergence rates guarantees on the excess risk ( i.e . O ( 1/T ) or O ( 1/C ) , where C is the total query budget ) . These are possible under a strong convexity assumption of the loss , specified below . Our contributions are the following . • As a preliminary , we revisit ( Section 3 ) the full information setting , with no limitations on queries . Maybe surprisingly , we contribute a new algorithm that is both simpler than existing ones and for which the proof of the fast convergence rate for excess risk is also elementary . Furthermore , for forecast we only need to consult 2 experts . The general principle of this algorithm will be reused in the limited observation settings . • We then investigate ( Section 4 ) the budgeted setting where we have a total query budget constraint C for the training phase ; then ( Section 5 ) the two-query setting where the agent is limited to m = 2 queries per training round . In both cases , we give precise efficiency guarantees on the number of training expert queries needed to achieve a given precision for forecast . The obtained bounds come both in instance-independent ( agnostic ) and instancedependent ( depending on the experts ’ structure ) flavors . • Finally , we give some lower bounds ( Section 6 ) were we show that fast rates can not be achieved if the agent is only allowed to consult one single expert per training round or for forecast . The following assumption on the loss will be made throughout the paper : Assumption 1 . ∀y ∈ Y : x ∈ X ⊆ Rd 7→ l ( x , y ) is L-Lipschitz and ρ-strongly convex . Recall that a function f : X → R is L-Lipschitz if ∀x , y ∈ X : |f ( x ) − f ( y ) | ≤ L‖x− y‖ , and ρ-strongly convex if the function : x→ f ( x ) − ρ 2 2 ‖x‖ 2 is convex . Remarks . Assumption 1 implies that the diameter of X is bounded by 8L/ρ2 and the quantity supx , x′∈X , y∈Y |l ( x , y ) − l ( x′ , y ) | is bounded by B : = 8L2/ρ2 ( this notation shorthand will be used throughout the paper ) . Consequently , without loss of generality we can assume that the loss is bounded by B ( see Lemma1 S-1 and subsequent discussion for details ) . It is satisfied , for example , in the following setting : least square loss l ( x , y ) = ( y − x ) 2 where x ∈ X and y ∈ Y with X and Y are bounded subsets of Rd . Prior knowledge on ρ is not necessary if L and an upper bound on the the l∞ norm of the target variable Y and the experts are known . 2 Discussion of related Work . Games with limited feedback ( slow rates ) : Our work investigates what happens between the full information and single-point feedback games . Learning with a restricted access to information was considered under various settings in [ 6 ] , [ 19 ] , [ 12 ] , [ 20 ] , [ 5 ] . A setting close to ours was considered in [ 21 ] , where the agent chooses in each round a subset of experts to observe their advice , then follows the prediction of one expert . To minimize the cumulative regret in the adversarial setting , they used an extension of the Exp3 algorithm , which allows to have an excess risk of O ( √ 1/T ) in the limited feedback setting and O ( √ log ( C ) /C ) in the budgeted case with a budget C. 1References starting with a prefix S- point to the supplemental material . The differences in the setting considered here is that ( a ) we are interested in the generalization error in the stochastic setting rather than the cumulative regret in an adversarial setting and ( b ) our assumptions of the convexity of the loss allow for the possibility of fast excess risk convergence . Moreover , we consider the more general case where the player is allowed to combine p out of K experts for prediction . The possibility of playing a subset of arms was considered in the literature of Multiple Play Multi-armed bandits . It was treated with a budget constraint in [ 26 ] for example ( see also [ 24 ] ) , where at each round , exactly p out of K possible arms have to be played . In addition to observing the individual rewards for each arm played , the player also learns a vector of costs which has to be covered with an a-priori defined budget C. In the stochastic setting , a UCB-type procedure gives a bound for the cumulative regret of O ( ∆−1min log ( C ) /C ) that holds only in expectation , where ∆−1min denotes the gap between the best choice of arms and the second best choice . This bound leads to an instance dependent bound of O ( √ log ( C ) /C ) in the worst case . In the adversarial setting , an extension of Exp3 procedure gives a bound of O ( √ log ( C ) /C ) for the cumulative regret that holds with high probability . In another online problem , where the objective is to minimize the cumulative regret in an adversarial setting with a small effective range of losses , [ 11 ] have shown the impossibility of regret scaling with the effective range of losses in the bandit setting , while [ 23 ] showed that it is possible to circumvent this impossibility result if the player is allowed one additional observation per round . However , in the settings considered , it is impossible to achieve a regret dependence on T better than the rate of O ( 1/ √ T ) . Fast rates in the full information setting : The learning task of doing as well as the best expert of a finite family in the sense of generalization error has been studied quite extensively in the full information case . In an adversarial setting , it is well-known that under suitable assumptions on the loss function ( typically related to strong convexity ) , an appropriately tuned exponential weighted average ( EWA ) strategy has cumulative regret bounded by the “ fast rate ” O ( log ( K ) /T ) [ 13 , 9 , 4 ] , which , combined with the online-to-batch conversion principle [ 8 , 4 ] ( also known as progressive mixture rule , [ 7 , 25 ] ) , yields a bound of the same order for the expected excess prediction risk in the stochastic case . However , it was shown that progressive mixture type rules are deviation suboptimal for prediction [ 2 ] , that is , their excess risk takes a value larger than c/ √ T with constant positive probability over the training phase . To lift the apparent contradiction between the two last statements , consider that the excess risk of the EWA can take negative values , since it is an improper learning rule . Thus negative and positive “ large ” deviations can compensate each other so that the expectation is small . The inefficiency of EWA in deviation is a significant drawback , and alternatives to the EWA progressive mixture rule that achieve O ( log ( K ) /T ) excess prediction risk with high probability were proposed by [ 17 ] and [ 3 ] . In [ 17 ] , the strategy consists in whittling down the set of experts by elimination of obviously suboptimal experts , and performing empirical risk minimization ( ERM ) over the convex combinations of the remaining experts . In [ 3 ] , the empirical star algorithm consists in performing an ERM over all segments consisting of a two-point convex combination of the ERM expert and any other expert . Note that the empirical star algorithm has the advantage that the final prediction rule is a convex combination of ( at most ) two experts . Linear regression with partially observed attributes : Other related work is that of [ 10 ] , and [ 14 ] on learning linear regression models with partially observed attributes . The most related setting to ours is the local budget setting , where the learner is allowed to output a linear combination of features for prediction . The key idea is to use the observed attributes in order to build an unbiased estimate of the full information sample , then to use an optimization procedure to minimize the penalized empirical loss . In our setting , the minimization of penalized empirical loss was shown to be suboptimal ( see [ 16 ] ) . Moreover , while we want to predict as well as the best expert , in [ 10 ] , the objective is to be as good as the best linear combination of features with a small additive term ( the optimal rate , in this case , is O ( 1/ √ T ) ) . Finally , we consider that the restriction on observed attributes ( experts advice ) does not apply only to the training samples but also to the testing data . Online convex optimization with limited feedback : The idea of using multiple point feedback to achieve faster rates appeared in the online convex optimization literature ( see [ 1 ] , and [ 22 ] ) . It was shown that in the setting where the adversary chooses a loss function in each round if the player is allowed to query this function in two points , it is possible to achieve minimax rates that are close to those achievable in the full information setting . The key idea is to build a randomized estimate of the gradients , which are then fed into standard first-order algorithms . These ideas are not convertible into our setting because we consider a non-convex set of experts .

Summary: The paper proposes a novel learning framework, which seems to me to be a cross between prediction with expert advice and bandit framework ported to stochastic off-line settings.  The key features are:  1. On every round, the learner chooses a limited number of experts to observe. 2. After T rounds, the learner outputs a function on experts' predictions that to be used on the next round. 3. The experts and the outcome are stochastic.   The framework seems to naturally fit a scenario where obtaining an expert prediction (e.g., by running a computationally expensive program) is costly.  The upper and lower bounds on the expectation of the learner's loss on the (T+1)th round are provided (lower bounds without proofs). The cases where m, the number of experts we are allowed to observe, is equal to the total and less than the total are treated separately.