7 Optimization techniques you may want to know in machine learning (Part 1)
Tags: Gradient Descent, Adam, Stochastic Gradient, Machine Learning, Optimization

Optimization is everywhere

Overview :

Introduction

1- Batch Gradient descent

2- Mini-Batch Gradient descent

3- Stochastic Gradient descent

4- Example of Gradient aggregation methods : SAGA

5- Example of diagonal scaling : RMSProp

6- Momentum

7- Adam

Introduction

In many machine learning problems, you try to minimize some objective funtion f with respect to some parameters x (1). This objective function is also called the cost function. This function often characterizes how well your machine learning algorithm is learning the mapping between your training examples and the desired outputs :

(1)

One classical problem is :

LINEAR REGRESSION :

(2) A : training data, y : corresponding labels

Given some matrix of features A composed of n examples as rows, d features as columns, and some label vector y composed of n real values corresponding to the labels of the n examples of A , we try to approach each value yi (ith value of y) by ai (ith row of A) multiplied by some vector x ; this vector x is learned by minimizing the cost function below :

(3) ||.|| is the L2-nom of a vector

A classical optimization algorithm to solve (3) is Gradient descent. In this article, we’ll explain different gradient descent techniques. We’ll also discuss the advantages and disadvantages of each one.

Here we assume that f is differentiable. We also assume that the solution of the optimization problem (1) exists. We add the assumption that f is a convex function in which case when we find a local minimum for f by a descent algorithm, It is necessary a global minimum. Convex functions are so easy to work with :

Convex vs Non convex function :

By the way, the function shown previously for linear regression is a convex function in the d dimensional space.

1- Batch Gradient descent

The algorithm for Batch GD in the general setting (1) is as follows :

(4) Gradient descent algorithm

It should return the value of our final iterate x_T which approximates the solution of the problem.∇f(x) denotes the gradient of f at the point x . It can be derived by computing partial derivatives of f with respect to each component of x :

One expression of the Gradient of f

It can be calculated using Taylor Formula, but we’re not going to dive into the details. Check this Wikipedia page for more insight : https://en.wikipedia.org/wiki/Gradient .

α _t is the stepsize at time t . You can choose It to be constant during the whole iterations or variable. This stepsize controls the speed of the convergence of the algorithm and by convergence we mean when the values of f stop decreasing from one iteration to the other meaning that we’ve reached our minimum. There is some interesting convergence analysis of Gradient descent according to the assumptions on f and to the choice of α\_t . If you’re interested, check these amazing notes of one of my teachers : https://mathematical-tours.github.io/book-sources/optim-ml/OptimML.pdf

The takeway idea is that you should pick a stepsize which is not too small so as you will slow down your convergence speed, and not too big so as the algorithm will diverge :

The intersections of the red arrow and the function’s graphic represent the values of f at successive iterations of Batch GD

Pros of Batch GD :

Batch Gradient descent works well in general. In addition, this algorithm was well-studied theoretically . There is a well established theory of convergence behind It .

Cons of Batch GD :

One iteration of Gradient descent requires computing the gradient of f using the whole training dataset, which might be computationally costly . Sometimes, all the training data might just not fit in the memory. Finally, Batch Gradient can be slow to converge for large datasets.

2- Mini-Batch Gradient descent