Complete Step-by-Step Gradient Descent Algorithm from Scratch
By: Albers Uzila
Tags: Gradient Descent, Programming, Hands On Tutorials, Machine Learning, Python

What is Optimization?

If you’ve been studying machine learning long enough, you’ve probably heard terms such as SGD or Adam. They are two of many optimization algorithms. Optimization algorithms are the heart of machine learning which are responsible for the intricate work of machine learning models to learn from data. It turns out that optimization has been around for a long time, even outside of the machine learning realm.

People optimize. Investors seek to create portfolios that avoid excessive risk while achieving a high rate of return. Manufacturers aim for maximum efficiency in the design and operation of their production processes. Engineers adjust parameters to optimize the performance of their designs.

This first paragraph of the Numerical Optimization book by Jorge Nocedal already explains a lot. It continues to state that even nature optimizes. Physical systems tend to a state of minimum energy. The molecules in an isolated chemical system react with each other until the total potential energy of their electrons is minimized. Rays of light follow paths that minimize their travel time.

To make sense of what optimization is, first of all, we must identify the objective, which could be the rate of return, energy, travel time, etc. The objective depends on certain characteristics of the system called variables. Our goal is to find values of the variables that optimize the objective. Often the variables are constrained, in some way.

Mathematically speaking, optimization is the process of maximizing or minimizing an objective function f(x) by searching for the appropriate variables x subject to some constraints cᵢ, which could be written compactly as follows.

where ℰ and ℐ are sets of indices for equality and inequality constraints, respectively.

This mathematical statement is definitely daunting at first sight, maybe because it is for the general description of optimization where not much could be inferred. But don’t worry, by the time we get to the code, everything will be clear.

Gradient Descent (the Easy Way)

Now, how do we solve the min function? Thanks to calculus, we have a tool called gradient. Imagine a ball at the top of a hill. We know that the hill has different slopes/gradients at different points. Due to gravity, the ball will move descent following the curve of the hill. Which way does it go? To the steepest gradient. After some time, the ball will reach a local minimum where the ground is relatively flat to its surroundings.

This is the nature of gradient descent. We could quantize the smooth path of the ball into tiny steps. At k-th step, we will have two quantities: the step length αₖ and the direction pₖ. To see gradient descent in action, let’s first import some libraries.

For starters, we will define a simple objective function f(x) = x² − 2x − 3 where x is real numbers. Since gradient descent uses gradient, we will define the gradient of f as well, which is just the first derivative of f, that is, ∇f(x) = 2x − 2.

Next, we define python functions for plotting the objective function and the learning path during the optimization process. What we mean by learning path is just points x after each descent step.

From the plot below, we could easily see that f has a minimum value at x = 1 (hence f(x) = -4). Let’s say we start at x = -4 (indicated by a red dot below), we will see whether gradient descent can locate the local minimum x = 1.

Image by Author

Define a simple gradient descent algorithm as follows. For every point xₖ at the beginning of step k, we maintain the step length αₖ constant and set the direction pₖ to be the negative of gradient value (steepest descent at xₖ). We take steps using the formula

while the gradient is still above a certain tolerance value (1 × 10⁻⁵ in our case) and the number of steps is still below a certain maximum value (1000 in our case).

Begin at x = -4, we run the gradient descent algorithm on f with different scenarios: