A Deep Insight into Optimization Algorithms
By: Bangxi Xiao
Tags: Gradient Descent

the fundamental concept for gradient descent algorithmn

Remember the time when we first dealt with an optimization problem in Statistics class, we tried to fit a straight line through a set of points, minimizing the summed vertical distance between the points and the line. The task is quite easy because the only thing we have to do is to calculate the first derivative of the squared distance and let it be 0, in which case one can solve for the optima symbolically. Later on, when got in touch with Logistic models, thought the optima parameters can still be expressed symbolically, we found the process of reaching optimal solution hader than the linear one. The above fact would make us thinking — what if, in some cases, the target function (also known as “the loss function” because we want to minimize the loss to get the optima) is so complex that solving for an expressible solution is almost impossible? This is when a bunch of optimization algorithms come in: they don’t get the answer direcrtly but in a iterative way — by iteratively tunning the parameters as they approach the optima.

Today, we are going to give a talk about two major optimization algorithm families — the Gradient Descent method as well as the Newton method. Both of the two approaches and their variants are implemented with an iterative manner.

Gradient Descent

Let’s start with a real-life scene: suppose you are placed on the top of a mountain (denote as the starting point) and you want to walk down to the bottom (denote as the optimal point), because we can only walk, there are also some rules regarding how we walk: in each step, you have to first choose a direction and then a step size, after which you can make the walk. The key idea is how to pick a proper direction? Since the only information we obtain from “the bottom” is that we need to go downward. Mathematically, simply it is the opposite direction of the gradient given the current point you stand, right? Now, we are cleared about the four elements mentioned above, let’s use math notation to expressed the process: let αα\alpha denote the step size, $

abla J(\Theta^{(t)})$ denote the downward direction at current position $\Theta^{(t)}$ (function $J(\Theta)$ describes the surface or more generally, the loss function we want to minimize), $\Theta^{(0)}$ denote the starting point and $\Theta^{(T)}$ denote the optimal point. Then the whole process can be describe as:

$$\Theta^{(t+1)} = \Theta^{(t)} +

abla J(\Theta^{(t)}),\quad t=0,1,\cdots,T$$

To be clear, $

abla$ indicates the opposite of a gradient while the gradient we solve from differentiation is upward.

Gradient Descent: A Toy Example

Suppose we have objective function $J(\theta)=\theta^{2}$ with starting point $x_{0}=(1,1)$ to minimize. Solving for the gradient of $J(\theta)=\theta^{2}$ we have $

abla J(\theta)=-2\theta$; also, we would like to set the step size a small value because instinctively thinking, a too-big step size might let you jump pass the optimal point, let step size $\alpha$ be $\alpha=0.4$. In all, the iterative process is as following:

$$\theta^{(t+1)}=\theta^{(t)}-0.8\theta^{(t)},\quad t=0,1,\cdots,T$$

Let’s go through first few steps to see what’s going on:

$$\theta^{(1)}=\theta^{(0)}-0.8\theta^{(0)}=1–0.8=0.2$$

$$\theta^{(2)}=\theta^{(1)}-0.8\theta^{(1)}=0.2–0.16=0.04$$

$$\theta^{(3)}=\theta^{(2)}-0.8\theta^{(2)}=0.04–0.032=0.008$$

$$\theta^{(4)}=\theta^{(3)}-0.8\theta^{(3)}=0.0016$$

As we can see, the $\theta$ value is getting closer ang closer to the optimal point $(0,0)$. We also simulated and visualize the process:

a toy example for gradient descent algorithm

With step size equals to 0.4, the gradient takes large steps when it starts; however, as getting closer to optimal, the convergence speed turns out to slow down — this is one of the major shortcuts of the gradient algorithms, but we will illustrate some variants to conquer the problem later.

Also, one might question that if the optimal we find is the global optimal — there is no guarantee that the result from a gradient descent algorithm is global optima. The conclusion is obvious if we take a closer look at the expression: when approaching bottom, the gradient at which could reduce to a very low value and hence the point might get stuck at that place, jumping left and right — never pumped out again. To address the above problem, one might set multiple starting points randomly and apply gradient descent on them to find the optimal points separately, by comparing the optimal values yielded from each point to finally make sure of the global optima.

Gradient Descent: A Multivariable Perspective

We will now illustrate a 3-Dimensional optimization task using gradient descent, the function is like:

$$J(Θ)=J(x,y)=sin(x)+cos(y)+\frac{x^{2}+y^{2}}{10}$$

Visualizing the objective function:

Noticing multiple critical points in the plot, it stresses the need to use different starting points. We set $5$ starting points in this case and the paths they travelled through are visualized as follows:

To check whether each starting point have managed to converge to the global optima, we can use convergence plot to identify.

convergence plot of different starting points

We surprisingly found that one of the paths fails to converge to the global optima but get stuck at a local minimal. Similar thing could also happen near the saddle point, using function $z=x^{2}-y^{2}$ as example. The white point $(x,y,z)=(0,0,0)$ in the plot below is the saddle point, which is either a local maximum point or a local minimum point.

getting out of the saddle point

We set 3 different starting points to optimize, 2 of which start from point with $y=0$ fail to get out of the saddle point; the green one however, managed to slide down, bypassing the saddle point.

Till now, we can conclude that one way to escape the saddle point is using multiple starting points, lowering the probability of being stuck. However, simply adding the number of starting points will not help if the number of dimension increases to a high value — the higher dimension space is so complex and unpredictable that it requires the number of starting points increased by geometric times — it is a computation disaster.

However, researchers have developed a various extended algorithms based on gradient descent to lower the computation cost as well as stop being trapped in local minima or saddle points.

Gradient Descent: Momentum

Imagine a ball, in the above example $z=x^{2}-y^{2}$, sliding from point $(x,y)=(6,0)$, gets stuck at the saddle point; however, if someone accidently touches the ball, applying a “momentum” directing to other direction to the ball, it gets out from the saddle point!

red path: the ball gets out of the saddle point by momentum

It is cleared that by adding momentum to the point, it can eventually jumping out of the saddle point (as shown as the red path in the above figure). The question is that how can we implement this mathematically?

Starting with a randomly initialized, direction-specified speed (vector), we use the vector to “influence” the gradient. Moreover, the speed vector in each iteration can be updated with following rule: if the gradient in current iteration has similar or same direction as the speed vector, the speed vector accumulates and grows; otherwise, it changes direction to the current gradient.

Specifically, we presume the following parameters for gradient descent with momentum: speed vvv, exponential growth rate $\beta$, learning rate αα\alpha and starting point $\Theta_{(0)}$. First, we should set the values for learning rate and exponential growth rate, empirically, use $0.01$ for learning rate and $0.9$ for exponential growth rate.

$$\beta=0.9,\quad\alpha=0.01$$

After initializing the speed vector and the starting point, in iteration ttt, do:

$$v^{(t+1)} = \beta v^{(t)} -(1-\beta)

abla J(\Theta^{(t)})$$

$$\Theta^{(t+1)}=\Theta^{(t)} + v^{(t+1)}$$

Applying the Gradient Descent with Momentum to optimize the above function results in:

Gradient Descent with Momentum: escaping the saddle point

With the help of momentum, the escape was a great success.

However, there is more benefits using momentum. Noticed from the update rule, “if the gradient in current iteration has similar or same direction as the speed vector, the speed vector accumulates and grows”, will this boosts the efficiency to find the optima? Let’s return to the multi local minima example:

$$J(\Theta)=J(x,y)=sin(x)+cos(y)+\frac{x^{2}+y^{2}}{10}$$

We optimize the above function with same starting points and examine the convergence plot of each:

Gradient Descent vs Gradient Descent with Momentum: convergence speed

As is shown in the above figure, after 2000 iterations, Gradient Descent algorithm is still running while Gradient Descent with Momentum has already met the convergence critera. This is because the accumulative effect of the gradient.

The downside of the algorithm is that the accumulated gradient vvv will get larger and larger as the algorithm runs, as a result of the fixed step size ratio (the $\alpha$), the step size will sometimes increase. The desired scene is that when iteration is going, the step size will decrease, to better approach the optimal point. To realize the goal, RMSprop, Adaptive Momentum and so on have been introduced. We will further analyze the algorithms in the following pages.

Gradient Descent: RMSprop and Adaptive Momentum (AdaM)

In order to reduce the learning rate αα\alpha as the gradient accumulates, we want learning rate be a function of the accumulated gradients — it decreases as accumulation grows. How to measure the accumulation? First thing that comes to mind must be the norm of the gradient and the truth is indeed so.

Specifically, we will first initialize a variable $r=[0.0,0.0]$ to record the accumulation as well as an exponential smoothing parameter $\rho,0<\rho<1$ to balance historical information and current information. The algorithm is described as following, initializing global learning rate αα\alpha, small constant ϵϵ\epsilon, in iteration ttt, we have:

$$g^{(t+1)}=

abla J(\Theta^{(t)})$$

$$r^{(t+1)} = \rho r^{(t)} + \frac{1-\rho}{d}\sum_i^d g_i^{(t+1)}*g_i^{(t+1)}$$

$$\Theta^{(t+1)}=\Theta^{(t)}+\frac{\alpha}{\sqrt{r^{(t+1)}+\epsilon}}g^{(t+1)}$$

According to the update process of RMSprop algorithm, imagine 2 scene: if the parameter space is relatively flat, the squared sum of gradients at each step will be small, resulting in a larger step size; if the parameter space is steep instead, the squared sum will be large, but resulting in a smaller step size. These good natures are very beneficial in accelerating the trainig process and improving the training accuracy. Also, using the example function $J(\Theta)=J(x,y)=sin(x)+cos(y)+\frac{x^{2}+y^{2}}{10}$ to see how fast the point converges under the RMSprop algorithm.

RMSprop: convergence plot (*sp is short for starting point)

The fastest point meets the convergence criteria in less than 90 iterations and the slowest takes no more than 400 steps. Comparing with basic Gradient Descent or Gradient Descent with Momentum, it’s really efficient!

Since both Gradient Descent with Momentum and RMSprop have such satisfying convergence speed, the 2 algorithm accelerate the convergence in 2 distinctive ways, why don’t we combine them together?

Yes, this is the Gradient Descent with Adaptive Momentum, or AdaM algorithm. It actively adjusts the momentum and accumulated squared sum gradients to pursue a better performance.

Initializing $\rho_1$ and $\rho_2$ as exponential decay rates for momentum and accumulated squared sum gradient respectively, $\Theta^{(0)}$ as the starting point, $m^{(0)}=[0.0,0.0]$ as the starting momentum vector and $r^{(0)}=[0.0,0.0]$ as the starting accumulation of squared sum gradient. Then, in iteration ttt we have the following parameters’ update:

$$g^{(t+1)}=

abla J(\Theta^{(t)})$$

$$m^{(t+1)}=\beta_1 m^{(t)}-(1-\beta_1) g^{(t+1)}$$

$$r^{(t+1)}=\beta_2 r + (1-\beta_2)g^{(t+1)}\odot g^{(t+1)}$$

$$m^{(t+1)}= \frac{m^{(t+1)}}{1-\beta_{1}^{t}}$$

$$r^{(t+1)} = \frac{r^{(t+1)}}{1-\beta_{2}^{t}}$$

$$\Theta^{(t+1)} = \Theta^{(t)} + \alpha \frac{m^{(t+1)}}{\sqrt{r^{(t+1)}+\epsilon}}$$

AdaM: Convergence plot

Using AdaM also indicates really fast convergence. However, the stability of AdaM algorithm compared with RMSprop or Gradient Descent with Momentum can’t be clearly revealed by such simple target function. We are going to go further in next part.

Now, we are going to apply these algorithms into objective functions constructed with real-world datasets.

Gradient Descent: Stochastic Gradient Descent and miniBatched Gradient Descent

In real-world applications, we use data to build model and construct loss function. Charging the optimization alrogithms into the loss function to minimize the loss, after the training, yields the optimal parameters. However, as the sample size grows, massive amount of data are plugged into the loss function, making the function more and more complex. AdaM, RMSprop and other variants of gradient descent algorithms can address the problem but only limitedly.

Another working way is to reduce the training size in each iteration and extend the number of iterations. This is when the miniBGD (mini-batched gradient descent) and SGD (stochasic gradient descent) involve in. Suppose we have a training dataset with size NNN:

$$\mathbf{D})_{(1)}^{(N)}=\{(X_1,y_1),(X_2,y_2),\cdots,(X_N,y_N)\}$$

Also a loss function $L$ with respect to $(X,y,\Theta)$:

$$L(X,y,\Theta)$$

In SGD, in each training iteration, we plug only 1 data point into loss function and do the gradient descent for NNN times:

$$J(\Theta)=L(\mathbf{D}_{(i)}^{(i)}, \Theta),i\in {1,2,\cdots,N}$$

In miniBGD, in each training iteration, we plug a batch of (say a size of $k$) data points (empirically we use $k=2^{n}$ as the size of minibatch) and do the gradient descent for $|\frac{N}{k}|$ times:

$$J(\Theta)=L(\mathbf{D}_{(kj-k))}^{(kj)},\Theta),j=1,2,\cdots,|\frac{N}{k}|$$

From above it is concluded when $k=1$, minibatched gradient descent is reduced to SGD and SGD can be considered as a special case of miniBGD.

Now we have introduced a variaty of extensions from gradient descent algorithm, in practice, the most often used is miniBGD combining with AdaM.

Gradient Descent: Applications

In this section, we will illustrate how gradient discent algorithm perfoms in complex models built with simulated as well as true data.

Starting with a linear model, the model has 20 parameters and in all 50,000 training points; the model will be trained with miniBatched Gradient Descent algorithm, detailed implementations include the application of AdaM algorithm.

As is known that a linear regression function takes the following form ($d$ is the number of features):

$$y_i = \sum_{k=1}^{d}x_{ik}w_{k}+\epsilon_{i},i\in \{1,2,\cdots,N\}$$

where $\epsilon_{i}, i\in \{1,2,\cdots,N\}$ is subjected to standard normal distribution. In this case, we want to minimize the distance between each data point and the fitted line, in which way we construct the loss function:

$$L(X,y,\hat{\Theta})=\frac{1}{N}\sum_{i=1}^{N}(\sum_{k=1}^{d}x_{ik}\hat{w}_{k}-y_i)^{2}$$

Solving for the gradient w.r.t. to $\Theta=\{w_{1}, w_{2}, \cdots, w_{d}\}$ and apply the AdaM algorithm:

Full-Batched AdaM: Mean-Squared Loss convergence plot

The above figure shows the convergence path of loss function we defined, using full-batched AdaM algorithm; however, the training process roughly takes us about 5 minutes. Here are also some convergence plots of the parameters:

Some weights’ convergence plots (Full-Batched AdaM)

When using miniBatched AdaM, the convergence plot might look unstable at the beginning, but eventually can meet their convergences.

miniBatched AdaM: Mean-Squared Loss convergence plot

Some weights’ convergence plots (miniBatched AdaM)

Surprisingly, the convergence process only takes less than 50 seconds, it is much faster than the full-batched one! That’s why in industry applications, miniBatched method is so widely used — it’s very computationally efficient!

Bangxi Xiao

Appendix Experiment Codes:

https://drive.google.com/file/d/1Ar4eQ11Pu0P8r2wcYMkTQQtoxzvnatfi/view?usp=sharing