Vectorization Of Gradient Descent - GeeksforGeeks

24 Oct, 2020
In Machine Learning, Regression problems can be solved in the following ways:
1. Using Optimization Algorithms – Gradient Descent
Batch Gradient Descent.
Stochastic Gradient Descent.
Mini-Batch Gradient Descent
Other Advanced Optimization Algorithms like ( Conjugate Descent ... )
2. Using the Normal Equation :
Using the concept of Linear Algebra.
Let’s consider the case for Batch Gradient Descent for Univariate Linear Regression Problem.
The cost function for this Regression Problem is :
Goal:
In order to solve this problem, we can either go for a Vectorized approach ( Using the concept of Linear Algebra ) or unvectorized approach (Using for-loop).
Here in order to solve the below mentioned mathematical expressions, We use for loop.
The above mathematical expression is a part of Cost Function.

The above Mathematical Expression is the hypothesis.

Code: Python Implementation of Unvectorzed Grad
# Import required modules.from sklearn.datasets import make_regressionimport matplotlib.pyplot as pltimport numpy as npimport time   # Create and plot the data set.x, y = make_regression(n_samples = 100, n_features = 1,                       n_informative = 1, noise = 10, random_state = 42)  plt.scatter(x, y, c = 'red')plt.xlabel('Feature')plt.ylabel('Target_Variable')plt.title('Training Data')plt.show()  # Convert y from 1d to 2d array.y = y.reshape(100, 1)   # Number of Iterations for Gradient Descentnum_iter = 1000   # Learning Ratealpha = 0.01   # Number of Training samples.m = len(x)   # Initializing Theta.theta = np.zeros((2, 1),dtype = float)   # Variablest0 = t1 = 0Grad0 = Grad1 = 0  # Batch Gradient Descent.start_time = time.time()   for i in range(num_iter):    # To find Gradient 0.    for j in range(m):        Grad0 = Grad0 + (theta[0] + theta[1] * x[j]) - (y[j])          # To find Gradient 1.    for k in range(m):        Grad1 = Grad1 + ((theta[0] + theta[1] * x[k]) - (y[k])) * x[k]    t0 = theta[0] - (alpha * (1/m) * Grad0)    t1 = theta[1] - (alpha * (1/m) * Grad1)    theta[0] = t0    theta[1] = t1    Grad0 = Grad1 = 0       # Print the model parameters.    print('model parameters:',theta,sep = '\n')   # Print Time Take for Gradient Descent to Run.print('Time Taken For Gradient Descent in Sec:',time.time()- start_time)  # Prediction on the same training set.h = []for i in range(m):    h.append(theta[0] + theta[1] * x[i])       # Plot the output.plt.plot(x,h)plt.scatter(x,y,c = 'red')plt.xlabel('Feature')plt.ylabel('Target_Variable')plt.title('Output')
 Output: 
model parameters:
[[ 1.15857049]
 [44.42210912]]
 
Time Taken For Gradient Descent in Sec: 2.482538938522339

Here in order to solve the below mentioned mathematical expressions, We use Matrix and Vectors (Linear Algebra).
The above mathematical expression is a part of Cost Function.

The above Mathematical Expression is the hypothesis.

Code: Python implementation of vectorized Gradient Descent approach
# Import required modules.from sklearn.datasets import make_regressionimport matplotlib.pyplot as pltimport numpy as npimport time   # Create and plot the data set.x, y = make_regression(n_samples = 100, n_features = 1,                       n_informative = 1, noise = 10, random_state = 42)  plt.scatter(x, y, c = 'red')plt.xlabel('Feature')plt.ylabel('Target_Variable')plt.title('Training Data')plt.show()    # Adding x0=1 column to x array.X_New = np.array([np.ones(len(x)), x.flatten()]).T  # Convert y from 1d to 2d array.y = y.reshape(100, 1)   # Number of Iterations for Gradient Descentnum_iter = 1000   # Learning Ratealpha = 0.01   # Number of Training samples.m = len(x)   # Initializing Theta.theta = np.zeros((2, 1),dtype = float)   # Batch-Gradient Descent.start_time = time.time()   for i in range(num_iter):    gradients = X_New.T.dot(X_New.dot(theta)- y)    theta = theta - (1/m) * alpha * gradients   # Print the model parameters.    print('model parameters:',theta,sep = '\n')   # Print Time Take for Gradient Descent to Run.print('Time Taken For Gradient Descent in Sec:',time.time() - start_time)  # Hypothesis.h = X_New.dot(theta) # Prediction on training data itself.   # Plot the Output.plt.scatter(x, y, c = 'red')plt.plot(x ,h)plt.xlabel('Feature')plt.ylabel('Target_Variable')plt.title('Output')
model parameters:
[[ 1.15857049]
 [44.42210912]]
 
Time Taken For Gradient Descent in Sec: 0.019551515579223633
Implementing a vectorized approach decreases the time taken for execution of Gradient Descent( Efficient Code ).Easy to debug.
Implementing a vectorized approach decreases the time taken for execution of Gradient Descent( Efficient Code ).
Easy to debug.
Machine Learning
Python
Machine Learning
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Comments
Old Comments
Difference between Informed and Uninformed Search in AI
Deploy Machine Learning Model using Flask
Support Vector Machine Algorithm
Types of Environments in AI
k-nearest neighbor algorithm in Python
Read JSON file using Python
Adding new column to existing DataFrame in Pandas
Python map() function
How to get column names in Pandas dataframe