Artificial Neural Network From Scratch Using Python Numpy
By: Madhav Mishra
Tags: Programming, Deep Learning, Artificial Intelligence, Data Science, Machine Learning

Finally, let’s build the ANN

ANN

So here we have:

Input node with some inputs (Real numbers; x1, x2, xn) with their weights (Real numbers; w1, w2, wn) and bias (Real number).

And this parameters (weights and bias) connects with our hidden nodes, where we compute weighted sum (sigma or z) for all inputs and theirs weight and then we apply a non-linear activation function (like, sigmoid, tanh, etc) and this generate a our final output (y).

Now, in our model we have 28 x 28 pixels of image (total pixels is 784 pixel) and this pixel is our inputs that goes to our input node then it goes to hidden node (single hidden layer) and then generate a output (single digits between 0 and 9).

Sigmoid Activation Function

Here our y-hat (or output of nodes) is sigmoid(dot product of weight and input ‘x’ + bias)

Implementing Sigmoid Activation Function:

#activation sigmoid

def sigmoid(x):

return 1. / (1.+np.exp(-x))

Cross-entropy Loss (a.k.a Cost, Error) Function

For ‘n’ classes and single samples or for ’n’ digits and single image, below we the have formula:

For ’n’ classes and single samples

But, for ’n’ classes and multiple(m) samples or for ’n’ digits and multiple single image, below we the have formula:

Implementing Cross-entropy Loss Function:

#cross-entropy for our cost function

def compute_multiclass_loss(Y, Y_hat):

L_sum = np.sum(np.multiply(Y, np.log(Y_hat)))

m = Y.shape[1]

L = -(1/m) * L_sum

return L

Back-propagation Using Gradient Descent Algorithm

Back-propagation

Back-propagation is just a way of propagating the total loss back into the neural network to know how much of the loss every node is responsible for, and subsequently updating the weights in such a way that minimizes the loss by giving the nodes with higher error rates lower weights and vice versa.

Gradient Descent

Gradient descent is a first-order iterative optimization algorithm for finding a local minimum of a differentiable function. To find a local minimum of a function using gradient descent, we take steps proportional to the negative of the gradient of the function at the current point.

Formula : new weight = prev. weight — learning rate*gradient

Gradient Descent Algorithm

Computing Gradient

Finally let’s implement it and train our model

n_x = X_train.shape[0]

n_h = 64

digits = 10

learning_rate = 1

epochs = 2000

Initializing Weights and bias:

W1 = np.random.randn(n_h, n_x)

b1 = np.zeros((n_h, 1))

W2 = np.random.randn(digits, n_h)

b2 = np.zeros((digits, 1))



X = X_train

Y = Y_train

Now, training start:

for i in range(epochs):

Z1 = np.matmul(W1,X) + b1

A1 = sigmoid(Z1)

Z2 = np.matmul(W2,A1) + b2

A2 = np.exp(Z2) / np.sum(np.exp(Z2), axis=0)



cost = compute_multiclass_loss(Y, A2)



dZ2 = A2-Y

dW2 = (1./m) * np.matmul(dZ2, A1.T)

db2 = (1./m) * np.sum(dZ2, axis=1, keepdims=True)



dA1 = np.matmul(W2.T, dZ2)

dZ1 = dA1 * sigmoid(Z1) * (1 - sigmoid(Z1))

dW1 = (1./m) * np.matmul(dZ1, X.T)

db1 = (1./m) * np.sum(dZ1, axis=1, keepdims=True)



W2 = W2 - learning_rate * dW2

b2 = b2 - learning_rate * db2

W1 = W1 - learning_rate * dW1

b1 = b1 - learning_rate * db1



if (i % 100 == 0):

print("Epoch", i, "cost: ", cost)



print("Final cost:", cost)

model loss

Generating our predictions and checking accuracy:

Z1 = np.matmul(W1, X_test) + b1

A1 = sigmoid(Z1)

Z2 = np.matmul(W2, A1) + b2

A2 = np.exp(Z2) / np.sum(np.exp(Z2), axis=0)



predictions = np.argmax(A2, axis=0)

labels = np.argmax(Y_test, axis=0)



print(confusion_matrix(predictions, labels))

print(classification_report(predictions, labels))

Okay, we got 92% accuracy which is pretty good.