Minimizing the cost function: Gradient descent | by XuanKhanh Nguyen | Towards Data Science

Imagine you are at the top of a mountain and want to descend. There may be many available paths, but you want to reach the bottom with a minimum number of steps. How can you come up with a solution? To answer that question, we will solve the gradient descent problem.
Gradient descent is one of the simplest algorithms that is used, not only in linear regression but in many aspects of machine learning. Several ideas build on this algorithm and it is a crucial and fundamental piece of machine learning.
The structure of this note:
Gradient descent
Apply gradient descent to linear regression
Gradient descent variants
A case study
This is a very long note. Grab a cup of coffee or tea and let’s get started.
A quick recap from my last note:
So we have our hypothesis function and we have a way of measuring how well it fits the data. We now need to estimate the parameters theta zero and theta one in the hypothesis function.
So here’s the problem setup. Assume that we have a function J, as theta zero, theta one. We want to minimize over theta zero and theta one of this function J(theta zero, theta one). And it turns out gradient descent is an algorithm for solving this general problem. We’re going to start with some initial guesses for theta zero and theta one. It doesn’t matter what they are, but a common choice would be we set theta zero to zero and set theta one to one. What we’re going to do in gradient descent is we’ll keep changing theta zero and theta one a little bit to try to reduce J(theta zero, theta one), until we wind at a minimum, or maybe at a local minimum.
Gradient descent is an efficient optimization algorithm that attempts to find a local or global minimum of the cost function.
A local minimum is a point where our function is lower than all neighboring points. It is not possible to decrease the value of the cost function by making infinitesimal steps.
A global minimum is a point that obtains the absolute lowest value of our function, but global minima are difficult to compute in practice.
Cost Function vs Gradient descent
We might argue that if the cost function and gradient descent are both used to minimize something then what is the difference and can we use one instead of the other?
Well, a cost function is something we want to minimize. For example, our cost function might be the sum of squared errors over the training set. Gradient descent is a method for finding the minimum of a function of multiple variables.
So we can use gradient descent as a tool to minimize our cost function.
Suppose we have a function with n variables, then the gradient is the length-n vector that defines the direction in which the cost is increasing most rapidly. So in gradient descent, we follow the negative of the gradient to the point where the cost is a minimum. In machine learning, the cost function is a function to which we are applying the gradient descent algorithm.
I assume that the readers are already familiar with calculus but will provide a brief overview of how calculus concepts relate to optimization here. So don’t worry friends, just stay with me... it’s kind of intuitive!
Machine learning uses derivatives in optimization problems. Derivatives are used to decide whether to increase or decrease the weights to increase or decrease an objective function. If we can compute the derivative of a function, we know in which direction to proceed to minimize it.
Suppose we have a function y = f(x) . The derivative f’(x) gives the slope of f(x) at point x. It specifies how to scale a small change in the input to obtain the corresponding change in the output. Let’s say, f(x) = 1/2 x2
We can reduce f(x) by moving in small steps with the opposite sign of the derivative. When f’(x) = 0,the derivative provides no information about which direction to move. Points where f’(x) = 0 are known as critical points.
The concept of convergence is a well defined mathematical term. It means that “eventually” a sequence of elements gets closer and closer to a single value. So what does it mean for an algorithm to converge? Technically what converges is not the algorithm, but a value the algorithm is manipulating or iterating. To illustrate this, let's say we are writing an algorithm that prints all the digits of pi.
Our algorithm starts printing numbers like:
x0 = 3.1x1 = 3.14x2 = 3.141x3 = 3.1415x4 = 3.14159...
As we can see, the algorithm prints increasing numbers close to pi. We say our algorithm converges to pi. And we call such functions convex functions (like a bowl shape). Now, let us consider the formula of gradient descent:
We implement this formula by taking the derivative (the tangential line to a function) of our cost function. The slope of the tangent line is the value of the derivative at that point and it will give us a direction to move towards. We make steps down the cost function in the direction with the steepest descent. The size of each step is determined by the parameter α (alpha), which is called the learning rate.
The learning rate determines the size of the steps that are taken by the gradient descent algorithm.
To reach a local minimum efficiently, we have to set our learning rate- parameter α appropriately, neither too high nor too low. Depending on where the initial point starts on the graph, it could end up at different points. Typically, the value of the learning rate is chosen manually, starting with 0.1, 0.01, or 0.001 as the common values.
In this case, gradient descent is taking too long to calculate; we need to increase the learning rate.
If our learning curve is just going up and down without reaching a lower point, we should try decreasing the learning rate.
Note:
If the learning rate is too big, the loss will bounce around and may not reach the local minimum.
If the learning rate is too small then gradient descent will eventually reach the local minimum but require a long time to do so.
The cost function should decrease over time if gradient descent is working properly.
How can we visualize this idea?
Let’s say we are at Mount Lyell (the highest point in Yosemite National Park), we hike down the hill following the path of the river. The job of gradient descent here is exactly what we aim to achieve— to reach the bottom-most point of the mountain. Mount Lyell is the data plotted in space, the surface representing the objective function, and the size of the step we move is the learning rate. The lowest point on the mountain is the value where the cost of the function reaches its minimum (the parameter α where our model presents more accuracy).
Assume also that Mount Lyell is shaped in such a way that the river will not stop at any place and will straightaway arrive at the foothill (like a bowl shape). In machine learning, we would have achieved our global minimum. However, straightforward optimization is not the case in real-life. The river may face a lot of pits on the way down. It might be trapped in the pits and fail to move downwards, which is a local minimum in machine learning.
When we are in a valley, there is no way we can descend the hill further. We can say we have converged. In machine learning, when gradient descent can’t reduce the cost function anymore and the cost remains near the same level, we can say it has converged to an optimum. The number of iterations for convergence may vary a lot.
The takeaway here is the initial values and learning rate. Depending on where we start at the first point, we could wind up at different local optima. Also, depending on the size of the step we take (learning rate) we might arrive at the foothill differently. These values are important in determining whether we will reach the foothill (global minima) or get trapped in the pits (local minima).
So we know gradient descent is an optimization algorithm to find the minimum of a function. How can we apply the algorithm to our linear regression?
To apply gradient descent, the key term here is the derivative.
Take the cost function and take a partial derivative with respect to theta zero and theta one, which looks like this:
Take the cost function and take a partial derivative with respect to theta zero and theta one, which looks like this:
To take the partial derivative, we hold all of the other variables constant. Let’s say, we want to take the partial derivative with respect to theta zero, we just treat theta one as a constant and vice versa.
But why do we use partial derivatives in the equation? So that we’ll have a way of measuring how well our hypothesis function fits the data. We need to estimate the parameters (theta zero and theta one) in the hypothesis function — that is, we want to know the rate of change value for theta zero and theta one. In calculus, partial derivatives represent the rate of change of the functions as one variable change while the others are held constant. We apply the partial derivatives with respect to theta zero and theta one to the cost function to point us to the lowest point.
2. Plug them back into our gradient descent algorithm
To find the best minimum, repeat steps to apply various values for theta zero and theta one. In other words, repeat the steps until convergence.
The process of finding the optimal values for theta zero and theta one is to then minimize our derivatives.
Hence, to solve for the gradient at the next step of the iteration, we iterate through our data points using our updated theta zero and theta one values and compute their partial derivatives. This new gradient tells us the slope of our cost function at our current position and the direction we should move to update our parameters. The size of our update is controlled by the learning rate.
Pros and cons of gradient descent
A simple algorithm that is easy to implement and each iteration is cheap; we just need to compute a gradient
However, it’s often slow because many interesting problems are not strongly convex
Cannot handle non-differentiable functions (biggest downside)
There are three types of gradient descent methods based on the amount of data used to calculate the gradient:
Batch gradient descent
Stochastic gradient descent
Mini-batch gradient descent
Batch Gradient Descent
In batch gradient descent, to calculate the gradient of the cost function, we calculate the error for each example in the training dataset and then take the sum. The model is updated only after all examples have been evaluated.
What if we have 1000 samples or in a worst-case scenario, one million samples? The gradient descent algorithm would need to run one million times. So batch gradient descent is not a good fit for large datasets.
As we see, batch gradient descent is not an optimal solution here. It requires a large number of computational resources, as the entire dataset needs to remain in memory. So, if we just need to move a single step towards the minimum, should we calculate the cost a million times?
Stochastic Gradient Descent (SGD)
In SGD, we use one training sample at each iteration instead of using the whole dataset to sum all for every step, that is — SGD performs a parameter update for each observation. So instead of looping over each observation, it just needs one to perform the parameter update.
Note: In SGD, before for-looping, we need to randomly shuffle the training examples.
SGD is usually faster than batch gradient descent, but its path to the minima is more random than batch gradient descent since SGD uses only one example at a time. But it’s ok as we are indifferent to the path, as long as it gives us the minimum and shorter training time.
SGD is widely used for larger dataset training, computationally faster, and can allow for parallel model training.
Mini-Batch Gradient Descent
Mini-batch gradient descent is a combination of both bath gradient descent and stochastic gradient descent.
Mini-batch gradient descent uses n data points (instead of one sample in SGD) at each iteration.
We have learned all we need to implement Linear Regression. Now it’s time to see how it works on a dataset. I have learned so much by implementing a simple linear regression in Python. I hope you will learn a thing or two after reading my note.
I downloaded the Boston weather reports from the National Oceanic and Atmospheric Administration. You can search on Kaggle for competitions, datasets, and other solutions. Our dataset contains information on weather conditions recorded on each day at a weather station. Information includes average temperature (TAVG), cooling degree days season to date (CDSD), extreme maximum temperature for the period (EMXT), heating degree days season to date (HDSD), maximum temperature(TMAX), minimum temperature (TMIN). In this example, we want to predict the maximum temperature taking input feature as the minimum temperature.
Let’s get our hands dirty with Python, shall we?
Import all the required libraries:
Import all the required libraries:
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as seabornInstancefrom sklearn.model_selection import train_test_splitfrom sklearn.linear_model import LinearRegression%matplotlib inline
2. Import the CSV dataset using pandas:
df = pd.read_csv(‘climate.csv’)df.dropna(inplace = True)
We use the dropna() function to remove missing values.
3. Check the number of rows and columns in our datasets.
df.shape
We should receive output as (903,9), which means our data contains 903 rows and 9 columns.
We can see the statistical detail of our dataset by using describe() function:
df.describe()
4. Visualize our dataset to see if we can manually find any relationship between the data.
fig,(ax1) = plt.subplots(1, figsize = (12,6))ax1.scatter (X, y, s = 8)plt.title (‘Min vs Max Temp’)plt.xlabel(‘TMIN’)plt.ylabel(‘TMAX’)plt.show()
5. Divide the data into “attributes” and “labels”.
Attributes are the independent variables while labels are dependent variables whose values are to be predicted. In our dataset, we only have two columns. We want to predict TMAX depending upon the TMIN recorded. Therefore our attribute set will consist of the “TMIN” column which is stored in the X variable, and the label will be the “TMAX” column which is stored in y variable.
X = df[‘TMIN’].values.reshape(-1,1).astype(‘float32’)y = df[‘TMAX’].values.reshape(-1,1).astype(‘float32’)
6. Split 80% of the data into the training set while 20% of the data go into the test set.
The test_size variable is where we specify the proportion of the test set.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
7. Train our algorithm.
For this, we need to import the LinearRegression class, instantiate it, and call the fit() method along with our training data.
h = LinearRegression()h.fit(X_train,y_train)print(h.intercept_) # to retrieve theta_0print(h.coef_) # to retrieve theta_1
The result should be approximately 16.25 for theta_0 and 1.07 for theta_1.
8. Make some predictions.
To do so, we will use our test data and see how accurately our algorithm predicts the percentage score.
y_pred = h.predict(X_test)compare = pd.DataFrame({‘Actual’: y_test.flatten(), ‘Predicted’: y_pred.flatten()})compare
As we can see from the table above, the predicted percentages are close to the actual ones. Let’s plot a straight line with the test data :
fig,(ax1) = plt.subplots(1, figsize = (12,6))ax1.scatter (X_test, y_test, s = 8)plt.plot(X_test,y_pred, color = ‘black’, linewidth = 2)\plt.show()
The predictions are pretty close to the actual plot, which indicates a small value of the variance.
9. Implement Linear Regression
#pick some random value to start withtheta_0 = np.random.random()theta_1 = np.random.random()def hypothesis (theta_0,theta_1,X):    return theta_1*X + theta_0def cost_function (X,y,theta_0,theta_1):    m = len(X)    summation = 0.0    for i in range (m):        summation += ((theta_1 * X[i] + theta_0) - y[i])**2    return summation /(2*m)def gradient_descent(X,y,theta_0,theta_1,learning_rate):    t0_deriv = 0    t1_deriv = 0    m = len(X)        for i in range (m):        t0_deriv += (theta_1 * X[i] + theta_0) - y[i]        t1_deriv += ((theta_1 * X[i] + theta_0) - y[i])* X[i]theta_0 -= (1/m) * learning_rate * t0_deriv    theta_1 -= (1/m) * learning_rate * t1_deriv        return theta_0,theta_1def training (X, y, theta_0, theta_1, learning_rate, iters):    cost_history = [0]    t0_history = [0]    t1_history = [0]        for i in range(iters):        theta_0,theta_1 = gradient_descent(X, y, theta_0, theta_1, learning_rate)        t0_history.append(theta_0)        t1_history.append(theta_1)        cost = cost_function(X, y, theta_0, theta_1)        cost_history.append(cost)          if i%10 == 0:              print ("iter={}, theta_0={}, theta_1={}, cost= {}".format(i, theta_0, theta_1, cost))return t0_history, t1_history, cost_history
We choose learning rate equals 0.01 for 2000 iterations, and plot our cost function J
t0_history, t1_history, cost_history = training (X, y, theta_0, theta_1, 0.01, 2000)#Plot the cost functionplt.title('Cost Function J')plt.xlabel('No. of iterations')plt.ylabel('Cost')plt.plot(cost_history)plt.ylim(ymin=0)plt.xlim(xmin=0)plt.show()
I found a cool way to visualize our data using Animations with Matplotlib. It takes 449 iterations for the model to come quite close to the best fit line.
import matplotlib.animation as animationfig = plt.figure()ax = plt.axes()# set up our plotplt.ylabel(“TMAX”)plt.xlabel(“TMIN”)plt.title(‘Linear Regression’)plt.scatter(X, y, color =’gray’,s =8)line, = ax.plot([], [], lw=2)plt.close()#Generate the animation data,def init():    line.set_data([], [])    annotation.set_text('')    return line, annotation# animation function.  This is called sequentiallydef animate(i):    #print(i)    x = np.linspace(-5, 20, 1000)    y = past_thetas[i][1]*x + past_thetas[i][0]    line.set_data(x, y)    annotation.set_text('Cost = %.2f e10' % (past_costs[i]/10000000000))    return line, annotationanim = animation.FuncAnimation(fig, animate, init_func=init, frames=np.arange(1,400), interval=40, blit=True)from IPython.display import HTMLHTML(anim.to_html5_video())
In this note, we studied the most fundamental machine learning algorithm — gradient descent. We implemented a simple linear regression with the help of the Scikit-Learning machine learning library. In the next note, we will focus on multiple linear regression.
The hardest part of any endeavor is the beginning, and you have passed that, so don’t stop!
Lucky for us, linear regression is well-taught in almost every machine learning curriculum, and there are a decent number of solid resources out there to help us understand the different parts of a linear regression model, including the mathematics behind. Below are some more resources if you find yourself wanting to learn even more.
Optimizing learning rateElimination of all bad local minima in deep learning (Cornell University)Elimination of all bad local minima in deep learning (MIT)What is convergence?Why Visualize Gradient Descent Optimization Algorithms?A case study — Moneyball — Linear RegressionSplit your data into training and testing (80/20)Partial derivative in gradient descent for two variablesFree data to work on: Lionbridge AI, Dataset Search
Optimizing learning rate
Elimination of all bad local minima in deep learning (Cornell University)
Elimination of all bad local minima in deep learning (MIT)
What is convergence?
Why Visualize Gradient Descent Optimization Algorithms?
A case study — Moneyball — Linear Regression
Split your data into training and testing (80/20)
Partial derivative in gradient descent for two variables
Free data to work on: Lionbridge AI, Dataset Search