Implementing SGD From Scratch. Custom Implementation of Stochastic... | by Jasmeet Singh | Towards Data Science

Before implementing Stochastic Gradient Descent let’s talk about what a Gradient Descent is.
Gradient Descent Algorithm is an iterative algorithm used to solve the optimization problem. In almost every Machine Learning and Deep Learning models Gradient Descent is actively used to improve the learning of our algorithm.
After reading this blog you’ll get to know how a Gradient Descent Algorithm actually works. At the end of this blog, we’ll compare our custom SGD implementation with SKlearn’s SGD implementation.
Pick an initial random point x0.x1 = x0 - r [(df/dx) of x0]x2 = x1- r [(df/dx) of x1]Similarly, we find for x0, x1, x2 ............. x[k-1]
Pick an initial random point x0.
x1 = x0 - r [(df/dx) of x0]
x2 = x1- r [(df/dx) of x1]
Similarly, we find for x0, x1, x2 ............. x[k-1]
Here r is the learning rate and df/dx is the gradient function to minimize our loss.
In Mini-Batch SGD the parameters are updated after computing the gradient of error with respect to a subset of the training set.
Let us take the Boston Housing Dataset from Kaggle as an example.
First, we will import all the necessary libraries.
import warningswarnings.filterwarnings("ignore")from sklearn.datasets import load_bostonfrom random import seedfrom random import randrangefrom csv import readerfrom math import sqrtfrom sklearn import preprocessingimport pandas as pdimport numpy as npimport matplotlib.pyplot as pltfrom prettytable import PrettyTablefrom sklearn.linear_model import SGDRegressorfrom sklearn import preprocessingfrom sklearn.metrics import mean_squared_errorfrom sklearn.model_selection import train_test_split
Now, we will load our dataset. Here, X contains the dataset we have and Y contains the label that we need to predict.
X = load_boston().dataY = load_boston().target
Remember to split your data before scaling to avoid the data leakage problem.
# split the data set into train and testX_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=0)
Use Standard Scalar function to standardize your dataset. Here, we only fit the train data because we don’t want our model to see this data before, so as to avoid overfitting.
scaler = preprocessing.StandardScaler()X_train = scaler.fit_transform(X_train)X_test = scaler.transform(X_test)
Creating the DataFrame using pandas.
X_train = pd.DataFrame(data = X_train, columns=load_boston().feature_names)X_train['Price'] = list(y_train)  X_test = pd.DataFrame(data = X_test, columns=load_boston().feature_names)X_test['Price'] = list(y_test)
Let’s see how our X_train looks like.
X_train.head()
Below is the loss function for our Linear Model that we need to minimize.
Now, we calculate the gradients for our loss function L w.r.t Weights(W) and Intercept (b). Following is the equations for calculating the gradients,
After calculating the gradients we keep changing our weights and intercept value with each iteration.
Finally, we’ll implement our SGD function.
def sgd_regressor(X, y, learning_rate=0.2, n_epochs=1000, k=40):        w = np.random.randn(1,13)  # Randomly initializing weights    b = np.random.randn(1,1)   # Random intercept value        epoch=1        while epoch <= n_epochs:                temp = X.sample(k)        X_tr = temp.iloc[:,0:13].values        y_tr = temp.iloc[:,-1].values                Lw = w        Lb = b                loss = 0        y_pred = []        sq_loss = []                for i in range(k):                          Lw = (-2/k * X_tr[i]) * (y_tr[i] - np.dot(X_tr[i],w.T) - b)            Lb = (-2/k) * (y_tr[i] - np.dot(X_tr[i],w.T) - b)                        w = w - learning_rate * Lw            b = b - learning_rate * Lb                        y_predicted = np.dot(X_tr[i],w.T)            y_pred.append(y_predicted)                loss = mean_squared_error(y_pred, y_tr)                    print("Epoch: %d, Loss: %.3f" %(epoch, loss))        epoch+=1        learning_rate = learning_rate/1.02            return w,b
We’ve kept our learning rate = 0.2, epochs = 1000 and batch size = 40. We can change the parameters accordingly to minimize our MSE.
We keep decreasing our learning rate by dividing it by 1.02, you can choose whatever value you want.
I’ve chosen MSE as my error metric, we can also choose RMSE.
Now, we create a predict function and compute our predicted values.
def predict(x,w,b):    y_pred=[]    for i in range(len(x)):        temp_ = x        X_test = temp_.iloc[:,0:13].values        y = np.asscalar(np.dot(w,X_test[i])+b)        y_pred.append(y)    return np.array(y_pred)w,b = sgd_regressor(X_train,y_train)y_pred_customsgd = predict(X_test,w,b)
from matplotlib.pyplot import figureplt.figure(figsize=(25,6))plt.plot(y_test, label='Actual')plt.plot(y_pred_customsgd, label='Predicted')plt.legend(prop={'size': 16})plt.show()print('Mean Squared Error :',mean_squared_error(y_test, y_pred_customsgd))
As we can see we got our MSE as 26.8 which is pretty decent.
from sklearn.linear_model import SGDRegressorclf = SGDRegressor(max_iter=1000, tol=1e-3)clf.fit(X_train, y_train)y_pred = clf.predict(X_test)
import matplotlib.pyplot as pltfrom matplotlib.pyplot import figureplt.figure(figsize=(25,6))plt.plot(y_test, label='Actual')plt.plot(y_pred, label='Predicted')plt.legend(prop={'size': 16})plt.show()print('Mean Squared Error :',mean_squared_error(y_test, y_pred))
SKlearn’s SGD implementation works much better than our custom implementation.
import matplotlib.pyplot as pltfrom matplotlib.pyplot import figureplt.figure(figsize=(25,6))plt.plot(y_pred, label='SGD')plt.plot(y_pred_customsgd, label='Custom SGD')plt.legend(prop={'size': 16})plt.show()print('Mean Squared Error of Custom SGD :',mean_squared_error(y_test, y_pred_customsgd))print("Mean Squared Error of SKlearn's SGD :",mean_squared_error(y_test, y_pred))
Well, our custom SGD did a pretty decent job compared to SKlearn’s and we could always do some hyper-parameter tuning to improve our custom model.
Implementing algorithms from scratch takes lot of time but if we use libraries then it still remains black box for us. Let me know in the comment section what you think about this.
Thank you for reading the blog, hope you got some insights on how a SGD actually works.