Multiclass classification Neural Network implementation using Numpy’s einsum | by ManiShankar Singh | Towards Data Science

We all start our machine learning journey by implementing a small classification or regression problem. Today I will implement a multiclass classification neural network using pure numpy. I will use the iris dataset. Below is the link to my code on colab notebook.
MulticlassClassificationNeuralNetwork
I will be creating a 2 layer neural network. First hidden layer will use leaky_relu activation function and second hidden layer will use softmax activation function.
Softmax function gives back probability of belonging to each class.
Derivative of softmax is a bit hairy to implement as it involves creating a Jacobian matrix. For all practical purposes, softmax will always be used in the outermost layer of neural network for multiclass classification and hence calculating its derivative is not important as long as we can get the derivative of cost function J with respect to logits z. Calculating this derivative is quite simple as compared to calculating the derivative of softmax function itself. I have shown the derivative in the image below:
If you are still persistent to get the derivative of softmax, its implementation is written in softmax_prime function shown in the picture below.
Pay special attention to softmax and binarize functions as shown above.
Now lets understand binarize function. It converts the labels 0,1,2,etc. into one hot encoded arrays like:
+=======+===========+| Label | Binarize |+=======+===========+| 0      | [1,0,0,0] |+ — — — -+ — — — — — -+| 1      | [0,1,0,0] |+ — — — -+ — — — — — -+| 2      | [0,0,1,0] |+ — — — -+ — — — — — -+| 3      | [0,0,0,1] |+ — — — -+ — — — — — -+
This is important because in multiclass classification the target will belong to one of the multiple classes present in data. Also the final softmax layer will return the probability of a record belonging to each of those class labels. So it is important to convert the class labels into one hot encoded record. In the implementation, binarize function is written in a clever way to utilize broadcasting concept of numpy.
I have used the dreaded and notorious as_strided function of numpy in the beginning to create a sliding window view of the array containing number of units in each layer. It is riskier to use this function because it does not check the boundaries of a given array while creating the view, which can easily lead to buffer overflow. Even though it is riskier to use, I have used it because why not. In deft hands it is like a swiss army knife. I wish tensorflow had something equivalent like as_strided function.
def murals_batch(units_count, x, y, lr, epochs, bias=False, _seed=42):    batch_size, ni = x.shape[-2:]    units_count.insert(0,ni)    units_count_arr = np.array(units_count)    L, = units_count_arr.shape # Number of layers + 1    # RED ALERT - `as_strided` function is like a LAND-MINE ready to explode in wrong hands!    arr_view = as_strided(units_count_arr, shape=(L-1,2), strides=(4,4))#     print(arr_view)    rng = np.random.default_rng(seed=_seed)    wghts = [None]*(L-1)    intercepts = [None]*(L-1)    # WEIGHTS INITIALIZATION    for i in range(L-1):        w_cols, w_rows = arr_view[i,:]        wghts[i] = rng.random((w_rows, w_cols))        if bias:            intercepts[i] = rng.random((w_rows,))    costs = np.zeros(epochs)    # Gradient Descent    for epoch in range(epochs):        # FORWARD PROPAGATION        # hidden layer 1 implementation, relu activation           h1a = np.einsum('hi,Bi -> Bh', wghts[0], x)        if bias:            h1a = h1a + intercepts[0]        h1 = relu(h1a)        # hidden layer 2 implementation, softmax activation        h2a = np.einsum('ho,Bo -> Bh', wghts[1], h1)         if bias:            h2a = h2a + intercepts[1]        hyp = softmax(h2a, _axis=1)        current_epoch_cost = -np.einsum('Bi,Bi', y, np.log(hyp))/batch_size#         print(current_epoch_cost)        costs[epoch] = current_epoch_cost        # BACKWARD PROPAGATION        # layer 2        dJ_dH2a = hyp - y        dJ_dW1 = np.einsum('Bi,Bj -> ij',dJ_dH2a, h1)/batch_size        # layer 1        dJ_dH1 = np.einsum('Bi,ij -> Bj', dJ_dH2a, wghts[1])        dJ_dH1a = dJ_dH1*relu_prime(h1a)        dJ_dW0 = np.einsum('Bi,Bj -> ij',dJ_dH1a, x)/batch_size        if bias:            dJ_dB1 = np.einsum("Bi -> i", dJ_dH2a)/batch_size            dJ_dB0 = np.einsum("Bi -> i",dJ_dH1a)/batch_size        # WEIGHTS ADJUSTMENT        wghts[1] = wghts[1] - lr*dJ_dW1        wghts[0] = wghts[0] - lr*dJ_dW0        if bias:            intercepts[1] = intercepts[1] - lr*dJ_dB1            intercepts[0] = intercepts[0] - lr*dJ_dB0    if bias:        return (costs, wghts, intercepts)    else:        return (costs, wghts)iris = load_iris()x = iris.datay = iris.target# NORMALIZEx_norm = normalize(x)x_train, x_test, y_train, y_test = train_test_split(x_norm, y, test_size=0.33, shuffle=True, random_state=42)# BINARIZEy_train = binarize(y_train)y_test = binarize(y_test)unit_per_layer_counts = [10,3]costs, fw, fb = murals_batch(unit_per_layer_counts, x_train, y_train, lr=0.01, epochs=19000, bias=True)plt.plot(costs)def predict(x,fw,fb):    h1a = np.einsum(’hi,Bi -> Bh’, fw[0], x)+fb[0]    h1 = relu(h1a)    h2a = np.einsum(’ho,Bo-> Bh’,fw[1],h1)+fb[1]    return softmax(h2a)
Also look at the fabulous time taken by the efficient code to execute 19000 epochs!!!
This efficiency is due to numpy’s Einstein summation function used to implement the neural network. I have discussed it in some details in my previous blog https://manishankar.medium.com/batch-gradient-descent-algorithm-using-numpy-einsum-f442ef798ee2
Please let me know your views about this blog post and code implementation with your comments.
I am a machine learning engineer at TCS and my (Digital Software & Solutions) team is building amazing products. Check out more about our products in the link below: