Low Complexity Approximate Bayesian Logistic Regression for Sparse Online Learning

1 INTRODUCTION . We consider online ( Bottou , 1998 ; Shalev-Shwartz et al. , 2011 ) binary logistic regression over a series of rounds t ∈ { 1 , 2 , . . . , T } . At round t , a sparse feature vector xt ∈ [ −1 , 1 ] d with dt d nonzero values , is revealed , and a prediction for the label yt ∈ { −1 , 1 } must be generated . The dimension d can be huge ( billions ) , but dt is usually tens or hundreds . Logistic regression is used in a huge portion of existing learning problems . It can be used to predict medical risk factors , to predict world phenomena , stock market movements , or click-through-rate in online advertising . The online sparse setup is also very common to these application areas , particularly , if predictions need to be streamed in real time as the model keeps updating from newly seen examples . A prediction algorithm attempts to maximize probabilities of the observed labels . Online methods sequentially learn parameters for the d features . With stochastic gradient methods ( Bottou , 2010 ; Duchi et al. , 2011 ) , these are weights wi , t associated with feature i ∈ { 1 , · · · , d } at round t. Bayesian methods keep track of some distribution over the parameters , and assign an expected mixture probability to the generated prediction ( Hoffman et al. , 2010 ; Opper & Winther , 1998 ) . The overall objective is to maximize a sequence likelihood probability , or to minimize its negative logarithm . A benchmark measure of an algorithm ’ s performance is its regret , the excess loss it attains over an algorithm that uses some fixed comparator values of w∗ 4 = ( w1 , w2 , . . . , wd ) T ( T denoting transpose ) . A comparator w∗ that minimizes the cumulative loss can be picked to measure the regret relative to the best possible comparator in some space of parameter values . Kakade & Ng ( 2005 ) ; Foster et al . ( 2018 ) ; Shamir ( 2020 ) demonstrated that , in theory , Bayesian methods are capable to achieve regret , logarithmic with the horizon T and linear with d , that even matches regret lower bounds for d = o ( T ) . Classical stochastic gradient methods are usually implemented as proper learning algorithms , that determine wt prior to observing xt , and are inferior in the worst-case ( Hazan et al. , 2014 ) , although , in many cases depending on the data , they can still achieve logarithmic regret ( Bach , 2010 ; 2014 ; Bach & Moulines , 2013 ) . Recent work ( Jézéquel et al. , 2020 ) demonstrated non-Bayesian improper gradient based algorithms with better regret . Unfortunately , superiority of Bayesian methods diminishes by their intractability . A theoretically optimal prior has a diagonal covariance matrix , with each component either uniformly or Normally distributed with large variance . Effects of such a prior can not be maintained in practical online problems with a large sparse feature set , as the posterior of such a prior no longer has the properties of the prior , but must be maintained as a subsequent prior . Gaussian approximations that rely on diagonalization of the covariance must be used . Neither normal nor the diagonal assumptions are true for the real posterior ( even with diagonal prior ) . They thus lead to performance degradations . Diagonalization is similar to linearization in convex optimization , specifically for stochastic gradient descent ( SGD ) ( Zinkevich , 2003 ) . It allows handling features independently , but limits performance . Bayesian learning literature focused on applying such methods to predict posterior probabilities , and provide model ( epistemic ) uncertainty measurements ( Bishop , 2006 ; Dempster , 1968 ; Huelsenbeck & Ronquist , 2001 ; Knill & Richards , 1996 ) . However , uncertainty of a feature is , in fact , mostly a function of the number of examples in which it was present ; a measure that can be tracked , not estimated . Methods , such as Variational Bayesian ( VB ) Inference ( Bishop , 2006 ; Blei et al. , 2017 ; Drugowitsch , 2019 ; Drugowitsch et al. , 2019 ; Ranganath et al. , 2014 ) , track such measurements by matching the posterior . However , as demonstrated in Rissanen ( 1984 ) seminal work , minimizing regret is identical to uncertainty reduction , as regret is merely a different representation of uncertainty . Regret can be universally minimized over the possible parameter space through a good choice of a prior . Hence , to minimize uncertainty , the role of an approximation is to preserve the effect of such a prior at least in the region of the distribution that dominates the ultimate posterior at the horizon T . This is a simpler problem than matching the posterior , and opens possibilities for simpler approximations that can lead to results identical to those of heavy methods as VB . VB methods are typically used offline to match a tractable posterior to the true one by upper bounding overall loss . They are computationally involved , requiring either iterative techniques ( Bishop , 2006 ; Blei et al. , 2017 ; Murphy , 2012 ) like Expectation Maximization ( EM ) ( Dempster et al. , 1977 ; Moon , 1996 ) ; or Monte Carlo ( MC ) sampling , replacing analytical expectation by an empirical one over a randomly drawn set . To converge , MC can be combined with gradient descent , either requiring heavy computations , or adapting stochastic gradient methods to update posteriors ( Broderick et al. , 2013 ; Knowles , 2015 ; Nguyen et al. , 2017a ; b ) . For online problems , the posterior of an example is the prior of the subsequent one . To minimize losses , online VB must converge to the optimal approximation at every example . Otherwise , additional losses may be incurred , as the algorithm may not converge at each example , while the target posterior keeps moving with subsequent examples . Moreover , combining methods that need to tune hyper-parameters defeats the parameter free nature ( Abdellaoui , 2000 ; Mcmahan & Streeter , 2012 ; Orabona & Pál , 2016 ) of Bayesian methods . Most Bayesian literature addressed the dense problem , where xt consists of mostly nonzero entries for every t , and the dimension d of the feature space is relatively small . Techniques , like Gaussian Mixtures ( Herbrich et al. , 2003 ; Montuelle et al. , 2013 ; 2014 ; Rasmussen , 2000 ; Reynolds et al. , 2000 ; Sung , 2004 ) , that may use VB , usually apply matrix computations quadratic in d on the covariance matrix . In many practical problems , however , a very small feature subset is present in each example . For categorical features , only one of the features in the vector is present at any example . Techniques , useful for the low dimensional dense problem , may thus not be practical . Paper Contributions : We provide a simple analytical Bayesian method for online sparse logistic and probit regressions with closed form updates . We generalize the method also for dense multidimensional updates , if the problem is not completely sparse . Our results are first to study regret for Bayesian methods that are simple enough to be applied in practice . They provide an example to the connection between uncertainty and regret , and more broadly the Minimum Description Length ( MDL ) principle ( Grunwald , 2007 ; Rissanen , 1978a ; b ; 1984 ; 1986 ; Shamir , 2015 ; 2020 ) . Empirical results demonstrate the advantages of our method over computationally involved methods and over other simpler approximations , both by achieving better regret and better loss on real data . As part of the algorithm , uncertainty measures are provided with no added complexity . We specifically demonstrate that it is sufficient to have an approximation focus on the location of the peak of the posterior and its curvature or value , which are most likely to dominate regret , instead of approximating the full posterior , which brings unnecessary complexity missing the real goal of preserving the effects of a good prior . In fact , approximating the full posterior may eventually lead to poor generalization and overfitting by focusing too much on the tails of the posterior . Our approach directly approximates the posterior , unlike VB methods that approximate by minimizing an upper bound on the loss . Finally , our approach leverages sparsity to solve a sparse problem . Related Work : The simplest single dimensional online logistic regression problem ( d = 1 and x1 , t = 1 , ∀t ) was widely studied . Jefferys ’ prior , ρ ( θ ) 4 = 1/ ( π √ θ ( 1− θ ) ) , is asymptotically optimal ( Clarke & Barron , 1994 ; Xie & Barron , 1997 ; 2000 ; Drmota & Szpankowski. , 2004 ) ) . It can be expressed in terms of log-odds weights w as ρ ( w ) = ew/2/ [ π ( 1 + ew ) ] . Applying a mixture leads to the Krichevsky & Trofimov ( 1981 ) ( KT ) add-1/2 estimator Q ( yt|yt−1 ) = [ nt−1 ( yt ) + 0.5 ] /t , where nt−1 ( yt ) counts occurrences of yt . We use yt to express a sequence from 1 to t. Applying this prior in a Follow The Regularized Leader ( FTRL ) setting ( McMahan , 2011 ) also leads to the KT estimator . This raised the question whether regret optimality generalizes to large dimensions ( McMahan & Streeter , 2012 ) . Hazan et al . ( 2014 ) showed that this was not the case for proper methods . The bounds , however , theoretically generalize for Bayesian methods ( Kakade & Ng , 2005 ; Foster et al. , 2018 ; Shamir , 2020 ) , with large variance Gaussian or uniform prior with diagonal covariance . Peaked priors fail , as for each feature in an example , other features provide a self excluding log-odds prior , that shifts the relation between the overall distribution and the feature weight . While wide priors are good theoretically , because of the intractability of the Bayesian mixture integrals , diagonal approximations that are used unfortunately degrade their effect . Bayesian methods have been studied extensively for estimating posteriors and uncertainty ( Bishop , 2006 ; Makowski et al. , 2002 ; Sen & Stoffa , 1996 ) . There is ample literature researching such techniques in deep networks ( see , e.g. , Blundell et al . ( 2015 ) ; Hwang et al . ( 2020 ) ; Kendall & Gal ( 2017 ) ; Lakshminarayanan et al . ( 2017 ) ; Malinin & Gales ( 2018 ) ; Wilson ( 2020 ) ) . Most of the work focuses on the ultimate posterior after the full training dataset has been visited . One attempts to leverage the uncertainty measurements to aid in inference on unseen examples . Techniques like expectation propagation ( EP ) ( Minka , 2001 ; 2013 ) ( see also Bishop ( 2006 ) ; Chu et al . ( 2011 ) ; Cunningham et al . ( 2011 ) ; Graepel et al . ( 2010 ) ) and VB are used to generate estimates of the posterior . In a dense setup , where there is a relatively small number features ( or units in a deep network ) , Gaussian Mixture models can also be learned , where a jointly Gaussian posterior is learned , usually with some kernel that is used to reduce the dimensionality of the parameters that are actually being trained . Such methods , however , do not fit the sparse online setup . Variational methods are derived utilizing Jensen ’ s inequality to upper bound loss of expectation by expectation of the negative logarithm of the product of the prior ρ ( w ) and data likelihood P ( yT |xT , w ) . Normalizing this joint by the expected label sequence probability gives the posterior P ( w|xT , yT ) . Then , a posterior Q ( · ) with a desired form is matched by minimizing the KL divergence KL ( Q||P ) , which decomposes into expectation w.r.t . Q ( ) ̇ over the loss on yT and KL ( Q||ρ ) between the approximated posterior and the true prior . The first term may require techniques like the iterative mean field approximation EM ( Bishop , 2006 ; Jaakkola & Jordan , 1998 ) , or MC sampling to be approximated . Gradient methods can also minimize KL ( Q||P ) . In the sparse setup , it is standard to assume a diagonal Q ( · ) . In an online setting , the process can be iterated over the examples ( or mini-batches ) , where the posterior at t is the prior at t+ 1 . Computing the approximate posterior may be very expensive if done for every example . SGD can be used with MC sampling , but that would incur additional losses , as the posterior changes between successive examples . Like VB , EP minimizes the opposite divergence KL ( P ||Q ) between the posterior and its approximate .

Summary: The authors propose a low complexity approximation method with closed analytic forms for doing logistic regression in the sparse, online setting. They first introduce the marginalized bayesian gaussian approximation approach, which essentially replaces the sigmoidal with a gaussian. They then give approximate expressions for the prediction and marginalization terms, as well as ways to approximate/update the posterior.