Sparsity in Online Learning with Lasso Regularization
By: Desh Raj
Tags: Online Learning, Machine Learning

Forward Backward Splitting (FOBOS)

Note: The method was named Forward Looking Subgradient (FOLOS) in the first draft and later renamed since it was essentially the same as an earlier proposed technique, the Forward Backward Splitting. The authors abbreviated it to FOBOS instead of FOBAS to avoid confusing readers of the first draft.

First, a little background. Consider an objective function of the form f(w) + r(w). In the case of a number of machine learning algorithms, the function f denotes the empirical sum of some loss function (such as mean squared error), and the function r is a regularizer (such as Lasso). If we use a simple gradient descent technique to minimize this objective function, the iterates would be of the form

where the g’s are vectors from the subgradient sets of the corresponding functions. From the paper:

A common problem in subgradient methods is that if r or f is non-differentiable, the iterates of the subgradient method are very rarely at the points of non-differentiability. In the case of the Lasso regularization function, however, these points are often the true minima of the function.

In other words, the subgradient approach will result in neither a true minima nor a sparse solution if r is the L1 regularizer.

FOBOS, as the name suggests, splits every iteration into 2 steps — a forward step and a backward step, instead of minimizing both f and r simultaneously. The motivation for the method is that for L1 regularization functions, true minima is usually attained at the points of non-differentiability. For example, in the 2-D space, the function resembles a Diamond shape and the minima is obtained at one of the corner points. Each iteration of FOBOS consists of the following 2 steps:

The first step is a simple unconstrained subgradient step with respect to the function f. In the second step, we try to achieve 2 objectives:

Stay close to the interim update vector. This is achieved by the first term. Attain a low complexity value as expressed by r. (Second term)

So the first step is a forward step, where we update the coefficient in the direction of the subgradient, while the second is a backward step where we pull the update back a little so as to obtain sparsity by moving in the direction of the non-differentiable points of r.

Using the first equation in the second, taking derivative w.r.t w, and equating the derivative to 0, we obtain the update scheme as

(Note: The equation above looks suspiciously similar to the Nesterov Accelerated Gradient (NAG) method for optimization. The authors have even cited Nesterov’s paper in related work. It might be interesting to investigate this further.)

This update scheme has 2 major advantages, according to the author.

First, from an algorithmic standpoint, it enables sparse solutions at virtually no additional computational cost. Second, the forward-looking gradient allows us to build on existing analyses and show that the resulting framework enjoys the formal convergence properties of many existing gradient-based and online convex programming algorithms.

In the paper, the authors also prove convergence of the method and show that on setting the intermediate learning rate properly, low regret bounds can be proved for both online as well as batch settings.