Mixability made efficient: Fast online multiclass logistic regression

1 Introduction . In online learning , a learner sequentially interacts with an environment and tries to learn based on data observed on the fly ( Cesa-Bianchi and Lugosi ( 2006 ) , Hazan et al . ( 2016 ) ) . More formally , at each iteration t ≥ 1 , the learner receives an input xt in some space X ; makes a prediction ŷt in a decision domain Ŷ and the environment reveals the output yt ∈ Y . The inputs xt and the outputs yt are sequentially chosen by the environment and can be arbitrary . No stochastic assumption ( except boundedness ) on the data sequence ( xt , yt ) 1≤t≤n is made . The accuracy of a prediction ŷt ∈ Ŷ at instant t ≥ 1 for the outcome yt ∈ Y is measured through a loss function ` : Ŷ × Y → R. The learner aims at minimizing his cumulative regret Rn ( f ) = n∑ t=1 ` ( ŷt , yt ) − n∑ t=1 ` ( f ( xt ) , yt ) , ( 1 ) with respect to any function f ∈ F in a reference class . All along this paper , we will consider parametric class of functions fθ indexed by θ ∈ Θ ⊂ Rd . We will also assume convexity of the loss according to index θ , which we denote ` x , y : θ 7→ ` ( fθ ( x ) , y ) for any ( x , y ) ∈ X × Y . In this 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . general context , many algorithms have been designed based on different assumptions and obtaining different trade-offs for regret . We review below the most relevant ones for our purpose . Assuming only convexity of the loss ` , a well-known strategy for the learner is Online Gradient Descent ( Zinkevich ( 2003 ) ) with an optimal regret of orderO ( √ n ) . However , if the loss is α-mixable , the learner may achieve the faster rate O ( 1α log ( n ) ) by using an aggregating forecaster ( see Vovk ( 2001 ) , Van Erven et al . ( 2015 ) ) . Yet , such an algorithm is not constructive in general and when it is , the computational complexity is often very high ( a notable exception is the least-squares setting ) . To reduce it , a stronger assumption has been introduced by Hazan et al . ( 2007 ) with η-exp concavity . Under the latter hypothesis , Online Newton Step ( ONS ) has a regret bounded by O ( 1η log ( n ) ) and a computational complexity of O ( n ) . However this efficiency comes at the price of deteriorating statistical performance . Indeed , η-exp concavity implies α-mixability for η ≥ α and in some cases , the gap between η and α can be very large . The most spectacular case of this phenomenon occurs for logistic regression . In this setting , the loss is defined as ` ( ŷt , yt ) = − log ( σ ( ŷt ) yt ) where σ ( z ) i = ezi∑ j e zj and the regret is computed with respect to linear functions F = { x 7→ Wx , W ∈ Θ } , where Θ = B ( RK×d , B ) is the Frobenius bounded ball of radius B > 0 . On this subset , the logistic loss is η-exp concave only for η ≤ e−B . The regret of Online Newton Step can thus be of order O ( eB log ( n ) ) . On the other hand , as remarked by Kakade and Ng ( 2008 ) ( binary case ) and Foster et al . ( 2018 ) , the logistic loss is 1-mixable and an aggregating algorithm may achieve O ( log ( nB ) ) . Nevertheless , the algorithm relies on Monte Carlo methods and must sample from log-concave probability distributions which is extremely computationally expensive ( see Table 1 ) . Therefore , in this framework , the exp-concavity assumption leads to an efficient algorithm with low dependence on B while mixability yields an inefficient algorithm with much better statistical performance . Between these two extremes , other algorithms and trade-offs have been analyzed . First , it has been shown that in some situations , Follow The Regularized Leader ( FTRL ) can achieve a fast rate without an exponential constant ( Bach et al . ( 2010 ) , Marteau-Ferey et al . ( 2019 ) , Ostrovskii et al . ( 2021 ) ) . However , several additional assumptions are necessary to achieve these rates . Unfortunately , they are essentially unavoidable . Indeed , Hazan et al . ( 2014 ) showed a polynomial lower bound for the proper algorithm ( i.e. , with linear prediction function ) in the regime B = log ( n ) . This prevents algorithms like ONS and Follow The Regularized Leader from reaching logarithmic regret without an exponential constant in B . This result motivates the search for improper algorithms with better computational complexity than Foster et al . ( 2018 ) . For binary logistic regression , two efficient improper algorithms have been proposed in the literature . First , in the i.i.d . framework , Mourtada and Gaïffas ( 2019 ) have proposed the Sample Minmax Predictor ( SMP ) . It achieves an excess risk of order O ( ( B2 + d ) /n ) with computational complexity equivalent to Follow The Regularized leader . In the online framework , Jézéquel et al . ( 2020 ) proposed AIOLI , which is based on quadratic approximations of the logistic loss as well as virtual labels to regularize . The regret is upper-bounded O ( dB log ( n ) ) and the computational complexity is O ( n ( d2 + log ( n ) ) ) . These previous works left open the question of achieving the same type of performance in a setting other than binary logistic regression . In particular , for multi-class logistic regression , no other algorithm than Foster et al . ( 2018 ) is known to achieve logarithmic regret without an exponential constant . Contributions We introduce in Section 3 a new generic online learning algorithm , that we call Gaussian Aggregating Forecaster ( GAF ) . GAF achieves logarithmic regret with a small multiplicative constant for a large class of convex loss functions ( see Theorem 1 ) . The latter includes several popular loss functions such as squared loss , binary and multi-class logistic loss . Our assumptions on the loss functions are slightly stronger than α-mixability but generally weaker than assumptions widely used in the statistical framework such as generalized self-concordance ( see Bach et al . ( 2010 ) ) . In the particular but significant setting of multi-class logistic regression , GAF has a regret bounded by O ( dKB2 + dKB log ( n ) ) and a total computational complexity of O ( nK3d2 + K2n4 ) , thus significantly improving on the O ( B6n25 ( Bn + dK ) 12 ) complexity of Foster et al . ( 2018 ) , which was the best known to date for algorithms without exponential dependence on B . Table 1 summarizes the regrets and computational cost obtained by the relevant algorithms in this framework . It is worth pointing out that , by using standard online-to-batch conversion Helmbold and Warmuth ( 1995 ) , this paper also provides new results on the excess risk in the statistical i.i.d . Indeed , even in this extensively used framework , the only existing improper algorithms without exponential constant in B so far are GAF and Foster et al . ( 2018 ) . Jézéquel et al . ( 2020 ) and Mourtada and Gaïffas ( 2019 ) are in fact restricted to binary outputs only . Our new algorithm is inspired by the aggregating forecaster of Vovk ( 2001 ) that we apply to quadratic approximations of the loss , in order to make it more efficient . The high-level idea is that these quadratic approximations replace complicated log-concave distributions with Gaussian distributions , from which it is much easier to sample and thereby drastically reducing the complexity of the algorithm . We believe that this approach can be applied to much broader contexts than those analyzed here . 2 Setting . We recall here the setting and introduce the main notations and assumptions that will be used throughout the paper . Setting and notation Our framework is formalized as a sequential game between a learner and an environment . At each forecasting instance t ≥ 1 , the learner is given an input xt ∈ X ; forms a prediction ŷt ∈ Ŷ ⊆ RK ( possibly based on the current input xt and on the past information x1 , y1 , . . . , xt−1 , yt−1 ) . Note that the prediction space Ŷ ⊆ RK may be uni-dimensional ( K = 1 ) in some settings ( least-square regression ) or multi-dimensional ( K ≥ 1 ) in some cases ( e.g. , vectorvalued regression , multi-class classification ) . Then , the environment chooses yt ∈ Y ; reveals it to the learner which incurs the loss ` ( ŷt , yt ) . Finally , the performance of the learner is assessed by the regret Rn ( θ ) = n∑ t=1 ` ( ŷt , yt ) − n∑ t=1 ` xt , yt ( θ ) , with respect to all θ ∈ Θ ⊂ Rd . Here , ` x , y ( θ ) = ` ( fθ ( x ) , y ) where F = { fθ : X → Ŷ } is the reference class of functions . For simplicity of notation , we also define ` t ( θ ) = ` xt , yt ( θ ) and Lt ( θ ) = ∑t s=1 ` s ( θ ) + λ‖θ‖2 for all t ≥ 1 and λ > 0 . In all specific examples considered in this work , the learner will be compared to linear functions fθ : x 7→ θ > Φ ( x ) , where θ ∈ Θ = B ( Rd , B ) and Φ is a function from X to Rd×K such that for any x ∈ X and i ∈ [ K ] , ‖Φ ( x ) . , i‖ ≤ R. We introduce and discuss below our main assumptions on the losses . Assumptions We assume that , for all ( x , y ) ∈ X × Y , the losses ` x , y are convex , C2 and satisfy the following assumptions . ( A1 ) The loss function ` is α-mixable . In other words , for all ( Gaussian ) probability distributions π over Rd and input x ∈ X , there exists ŷ ∈ Ŷ such that for all y ∈ Y , ` ( ŷ , y ) ≤ − 1 α log ( Eθ∼πe−α ` x , y ( θ ) ) . ( A2 ) There exists ζ > 0 such that , for all ( x , y ) ∈ X × Y and θ1 , θ2 ∈ Rd , ` x , y ( θ1 ) ≤ ` x , y ( θ2 ) +∇ ` x , y ( θ2 ) > ( θ1 − θ2 ) + eζ‖θ1−θ2‖ 2 ‖θ1 − θ2‖2∇2 ` x , y ( θ2 ) . ( A3 ) There exists β > 0 such that , for all ( x , y ) ∈ X × Y and ( θ1 , θ2 ) ∈ Θ× Rd , ` x , y ( θ1 ) ≥ ` x , y ( θ2 ) +∇ ` x , y ( θ2 ) > ( θ1 − θ2 ) + β 2 ( θ1 − θ2 ) > ∇2 ` x , y ( θ2 ) ( θ1 − θ2 ) . ( A4 ) ` is γ-smooth i.e. , for all x ∈ X , y ∈ Y , θ ∈ Rd , ∇2 ` x , y ( θ ) ≤ γI . In general , these assumptions are rather weak and related to standard assumptions of ( online ) convex optimization . ( A1 ) follows from instance from exp-concavity . As we illustrate in Section 4 , all are satisfied for broadly used loss functions such as squared loss , binary and multi-class logistic loss . We provide more details on these assumptions below . Mixability ( A1 ) is a standard assumption in online learning to achieve fast rates for the regret ( Van Erven et al. , 2015 ) , which was introduced by Vovk ( 2001 ) . It is a weaker condition than exp-concavity because ( A1 ) holds with ŷ = Eθ∼π [ θ ] for η-exp concave loss functions when η ≥ α . Assumption ( A2 ) basically prevents the Hessian from changing too quickly on θ . This condition is new but not very restrictive because , as we prove in Lemma 6 , it is weaker than other widely accepted assumptions such as generalized self-concordance ( Bach et al. , 2010 ) . The lower bound ( A3 ) is the most original assumption and is crucial to make aggregating forecaster efficient . This bound is close to the one needed in the analyses of ONS , which can be derived from η-exp-concavity . However , the key difference is that ( A3 ) uses the Hessian in the quadratic term and not the outer product of the gradient . This makes the right-hand-side closer to the third Taylor series approximation of the loss and allows much larger values for β . For example , as we will show in the section 4.2 , this change allows us to remove the exponential constant for the logistic regression setting . In this case , ( A3 ) is verified with β ' B−1 , while η-exp-concavity would require η ' e−B .

Summary: With online learning for multiclass logistic regression, its regret has a trade-off against the computational cost. For example, ONS can be computed in $O(n)$ time ($n$: number of instances) while the regret is $O(e^B \log n)$: exponentially increased against the radius of regret computation $B$. [Foster 2018] can provide the regret $O(\log(Bn))$ but requires $O(n^{37})$ time. The proposed method can provide another trade-off between them, and achieved the regret $O(B^2 + B\log n)$ with the computational cost of $O(n^4)$. This is much faster than [Foster 2018] with acceptable computational cost.