BFGS in a Nutshell: An Introduction to Quasi-Newton Methods
By: Adrian Lam
Tags: Getting Started, Editors Pick, Machine Learning, Optimization, Mathematics

If this is your first time learning about Newton’s method, you’re probably just as shocked as I was when it was my first time. “Surely this is too good to be true!” you might exclaim. Why ever use gradient descent if Newton’s method converges to the minimum so much faster?

It turns out that the benefits we gain from Newton’s method comes at a cost. There are two main issues with Newton’s method:

It is sensitive to initial conditions. This is especially apparent if our objective function is non-convex. Unlike gradient descent, which ensures that we’re always going downhill by always going in the direction opposite the gradient, Newton’s method fits a paraboloid to the local curvature and proceeds to move to the stationary point of that paraboloid. Depending on the local behavior of our initial point, Newton’s method could take us to a maximum or a saddle point instead of a minimum. This problem is exacerbated in higher dimensional non-convex functions, where saddle points are much more likely to occur compared to minimum points. We thus see that Newton’s method is really only appropriate for minimizing convex objective functions. The silver lining here, though, is that a considerable amount of optimization problems in machine learning are convex problems, such as linear (and ridge) regression, logisitic regression, and support vector machines. An obvious example of a non-convex model would be neural networks. For the rest of this article, we will restrict our attention to only convex objective functions. Correspondingly, we require the Hessian to be positive definite. Newton’s method is very computationally expensive. While the computation of the gradient scales as O(n), the computation of the inverse Hessian scales as O(n³) (computing the Hessian scales as O(n²), inverting it scales as O(n³)). As the dimensions of our problem increases, the overhead in memory and time gets out of hand very quickly. For example, in 50 dimensions, we’ll have to calculate 50(50+1)/2 = 1275 values for the Hessian at each step, and then perform approximately another 50³ operations to invert it. It’s clear at this point that the benefit of an increased convergence rate will be far outweighed by the large cost of the additional computation time.

Despite its limited practical use, Newton’s method is still of great theoretical value. We did see how efficient the second-order optimization can be if used correctly. What if we could somehow leverage the efficiency gained from considering second-order behavior, but avoid the computational cost of calculating the inverse Hessian? In other words, can we get a sort of hybrid between gradient descent and Newton’s method, where we can have faster convergence than gradient descent, but lower operational cost per iteration than Newton’s method?

It turns out there’s a class of optimization methods, called quasi-Newton methods, that does just that.

Quasi-Newton methods

We went through Newton’s method for optimization, which, in contrast to vanilla gradient descent, leverages second-order behavior in addition to first-order behavior at each step, making for a much faster convergence to the minimum. However, we also saw the downsides to Newton’s method — one of which is how computationally expensive it is to both calculate the Hessian and to invert it, especially when dimensions get large. Quasi-Newton methods are a class of optimization methods that attempt to address this issue.

Recall that in Newton’s method, we make the following update at each iteration:

where the Hessian is computed and inverted at each step. In quasi-Newton methods, instead of computing the actual Hessian, we just approximate it with a positive definite matrix B, which is updated from iteration to iteration using information computed from previous steps (we require B to be positive definite because we are optimizing a convex function, and this automatically takes care of the symmetry requirement of the Hessian).We immediately see that this scheme would yield a much less costly algorithm compared to Newton’s method, because instead of computing a large amount of new quantities at each iteration, we’re largely making use of previously computed quantities.

At this stage the nature of the update scheme for B has been left vague, because the specific update for B is given by the specific quasi-Newton method used. There is however one condition that all quasi-Newton methods must share, and that is the Hessian approximation B must satisfy the quasi-Newton condition (or secant equation):

which is obtained from the first order Taylor expansion of ∇ f(xᴋ₊₁) about ∇f(xᴋ) (we can also view this as sort of a finite difference equation of the gradient itself). We can rewrite the quasi-Newton condition more succinctly by letting yᴋ = ∇ f(xᴋ₊₁) −∇f(xᴋ) and Δxᴋ = xᴋ₊₁−xᴋ, so that we have

Furthermore, we can verify the positive definiteness of our Hessian approximation B by pre-multiplying the quasi-Newton condition with Δxᴋᵀ, so our requirement for positive definiteness (ie. the curvature condition) can be expressed as Δxᴋᵀ yᴋ>0.

Before we go further however, let’s take a step back and consider only one dimension, where our intuition is strong. The secant equation, in one dimension, is

and the curvature condition is satisfied by requiring

(f′ᴋ₊₁−f′ᴋ)/(xᴋ₊₁−xᴋ) > 0. Solving for f″ and substituting into Newton’s method in one dimension, we get

We thus have here an optimization method that leverages the (approximate) second-order behavior of the objective function in order to converge faster, without actually taking any second derivatives. Instead, at each iteration k+1, we construct an approximate inverse second derivative using only quantities from previous steps — in this case the first derivative from the previous two iterations k and k−1.

This method — approximating Newton’s method in one dimension by replacing the second derivative with its finite difference approximation, is known as the secant method, a subclass of quasi-Newton methods.

Now back to our n-dimensional quasi-Newton condition (equation 4). At first glance, it looks like we could just work analogously to our one-dimensional case: we can simply solve for Bᴋ₊₁ directly, and substitute it into Newton’s iterative step (2). Job done, right?

Not quite, actually. Despite looking deceptively similar to our one-dimensional case, remember that B is in general a symmetric n × n matrix, with n(n+1)/2 components, whereas our equation only has n components. This means that we’re trying to solve for n(n+1)/2 unknowns with only n equations, making this an underdetermined system. In fact, we were only able to find a unique solution to the secant equation in one dimension because the unknown components of the Hessian, being 1(1+1)/2 = 1, coincide with the one equation that we have. In general, there are n(n+1)/2 − n= n(n−1)/2 unknowns that we cannot solve for.

This is where quasi-Newton methods come in, where the secant method is generalized to multidimensional objective functions. Instead of approximating the second derivative merely by using the finite difference like in the secant method, quasi-Newton methods have to impose additional constraints. The common theme still runs through though — at each iteration k+1, the new Hessian approximation Bᴋ₊₁ is obtained using only previous gradient information.

Various quasi-Newton methods have been developed over the years, and they differ in how the approximate Hessian B is updated at each iteration. As of today, the most widely used quasi-Newton method is the BFGS method, and this will be our focus for the remaining of this article.

BFGS optimization

It’s been somewhat of a long trek so far, so let’s pause for moment and do a quick recap before moving on. Our objective is to find the minimum of a (twice-differentiable) convex function. A simple approach to this is gradient descent — starting from some initial point, we slowly move downhill by taking iterative steps proportional to the negative gradient of the function at each point. Newton then taught us that we can take far less steps and converge much quicker to the minimum if we also consider the second-order behavior of the function, by computing the (inverse) Hessian at each step. This comes at a cost, however, as calculating (and inverting) the Hessian takes a lot of resources. A compromise would be to instead just approximate the Hessian at each step, subject to the quasi-Newton condition. In one dimension, this just amounts to approximating the second derivative by replacing it with a finite difference approximation; this method of optimization is called the secant method. In more than one dimension, the quasi-Newton condition does not uniquely specify the Hessian estimate B, and we need to impose further constraints on B to solve for it. Different quasi-Newton methods offer their own method for constraining the solution. Here, we will focus on one of the most popular methods, known as the BFGS method. The name is an acronym of the algorithm’s creators: Broyden, Fletcher, Goldfarb, and Shanno, who each came up with the algorithm independently in 1970 [7–10].

Figure 2. From left to right: Broyden, Fletcher, Goldfarb, and Shanno.

We saw previously that in n>1 dimensions, the quasi-Newton condition (4) is underdetermined. To determine an update scheme for B then, we will need to impose additional constraints. One such constrain that we’ve already mentioned is the symmetry and positive-definiteness of B — these properties should be preserved in each update. Another desirable property we want is for Bᴋ₊₁ to be sufficiently close to Bᴋ at each update k+1. A formal way to characterize the notion of “closeness” for matrices is the matrix norm. Thus, we should look to minimize the quantity ||Bᴋ₊₁−Bᴋ||. Putting all our conditions together, we can formulate our problem as